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CHAPTER I 



INTRODUCTION 

Intriguing capabilities for computer-assisted instruction 
(CAI) have been developed during the past decade. CAI course mate- 
rial for studerts can be prepared to take advantage of the CAI system 
capabilities and tc produce dynamic student-computer interactions. 
Such CAI course preparation is a time-consuming task involving course 
design, development, and evaluation. 

Evaluation traditionally has been viewed as an activity to 
follow course completion. Authorities have recently indicated, how- 
ever, that carefully applied, systematic evaluation may enhance the 
quality of a CAI course when it is concurrent with course design and 
development. 

This study examined procedures for making evaluation an inte- 
gral part of course design and development. The investigator conduc- 
ted the study while she served as a course author for the preparation 
of a CAI course entitled "Education of Visually Handicapped Children." 

A. Statement of the Problem 
The purpose of this study was to investigate and to report an 
author's use of formative evaluation during the preparation of a 
course designed for present^ . . '.nts via computer-assisted 

instruction. The activities u ..ip.ef^ea iu'ing the study were planned 
to answer the following ques^ 

1 



1. What models for fcmative evaluation are reported in the 
literature related to the development of course materials? 

2. Of the models examined, which model is appropriate for 
an author to use during the preparation of a CAI course? 

3. VJhat are the results of using the selected model for 
formative evaluation during the preparation of CAI course 
materials? 

4. What information, procedures, and forms can be identified 
for possible inclusion in a manual for other authors to 
use in the formative evaluation process in the pre- 
paration of instructional programs? 

B. General Plan of Study Report 
In Chapter II the investigator will report the review of the 
literature pertaining to -nodels for formative evaluation. Chapter 
III will include the rationale for the selection of a particular model 
for formative evaluation and will outline the procedures used to 
carry out the steps of that formative evaluation model during the 
development of a CAI course. The results of the formative evaluation 
program will be summarized in Chapter IV along with procedures, forms, 
and information suitable for other CAI course authors to use in 
formative evaluation. The conclusions, limitations, suggestions for 
further investigation, and summary of the study will be presented in 
Chapter V. 



CHAPTER II 



REVIEW OF THE LITERATURE 

The review of the literature examines three main areas: 
evaluation of instructional programs, computer-assisted instruction 
(CAI), and evaluation of CAI course material during initial prepara- 
tion. A distinction is made between formative and summative 
evaluation. 

A. Evaluation of Instructional Programs 
1. Formative and Summative Evaluation 
Authorities have referred to Cronbach's paper entitled "Course 
Improvement Through Evaluation" presented in 1963 as a "classic."^ 
In this paper Cronbach defined evaluation as the "... collection 
and use of information to make decisions about an educational pro- 
gram. He indicated that such information could be used for course 
improvement, for decisions about individual students, or for 
administrative regulations. Cronbach emphasized the importance of 
evaluation for the purpose of course improvement: 



Gene Glass, "Design of Evaluation Studies" (paper presented 
at the Council for Exceptional Children Special Conference on Early 
Childhood Education, New Orleans, December, 1969). 



Lee J. Cronbach, "Course Improvement Through Evaluation." 
Teachers College R ecord. LVII (May, 1963), p. 672. 
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The greatest service evaluation can perform 1s to Identify 
aspects of the course where revision is desirable. . . . 
To be influential in course improvement, evidence must 
become available midway in curriculum development, not in 
the home stretch, when the developer is naturally reluctant 
to tear open a supposedly finished body of materials and 
techniques. Evaluation, used to improve the course while 
it is still fluid, con^ -ore to improvement of edu- 

cation than evaluatiOL appraise a product already 

placed on the market.' 

Cronbach stated that the analysis of performance on single test items 
or the record of responses to different typas of problems could be 
more informative than an analysis of total scores. He viewed evalua- 
tion as: 



... a fundamental part of curriculum development, not 
an appendage. Its job is to collect facts the course 
developer can and will use to do a better job, and facts 
from which a deeper understanding of the educational pro- 
cess will emerge. 2 

3 

Scnven proposed using the terms "formative" and "summative" 

to distinguish between evaluation to improve an instructional program 

or curriculum during its development and evaluation to determine the 

worth or effectiveness of an instructional program once it had been 

completed. He suggested that, in order to avoid potential clashes 

between curriculum writers and professional evaluators, 

. « . formative evaluators should, if at all possible 
be sharply distinguished from the summative evaluators, 
with whom they may certainly work in developing an 



^Ibid., p. 675. 
^Ibid., p. 683. 



3 

Michael Scriven, "The Methodology of Evaluation," in 
Perspectives of Curriculum Evaluation , AREA Monograph Series on Cur- 
riculum Evaluation, No. 1 (Chicago: Rand McNally and Company, 1967), 
p. 43. 



acceptable surnmatlve evaluation schema, but the formative 
evaluators should Ideally exclude themselves from the 
role of judge In the summatlve evaluation.^ 

Scrlven maintained that in the early stages of any kind of 

curriculum project general objectives or goals are formulated. These 

goals, which should not be considered absolute commitments but rather 

reminders subject to alteration, might range from motivational and 

cognitive goals to the goal of producing a marketable program. 

Scrlven declared that these goals were to be themselves Items for 

evaluation; performance measured against goals was not to be the only 

concern of the evaluator. To him It was "... obvious that If the 

goals aren't worth achieving then it is uninteresting how well they 

2 

are achieved." 

Scrlven outlined three types of activities which could 
facilitate both the evaluation of the goals and the evaluation of 
performance measured against those goals. These activities are: 

1. Regular reexamination and modification of proposed gen- 
eral objectives or goals of the project. 

2. Construction of a test-question pool, which thus becomes 
an "operational version of the goals"^ and as such, also 
requires regular reexamination and modification in light 
of any changes in the project goals. 

3. External judgments about the consistency of the project 
goals, content, and test-question pool. 



Ibid . , p. 45», ^ Ibid . , p. 52,, ^ Ibid ., p. 56. 



Scriven saw several refinements of the above activities as crucial to 
formative evaluation studies since they could uncover the causes of 
poor results: 

Essentially, we need to know about the success of three con- 
nected matching problems: first, the match between goals 
and course content; second, the match between goals and 
examination content; third, between course content and 
examination content. . . Only in this way are we likely 
to be able to track down the real source of disappointing 
results. 1 

Stolurow in a paper presented at a Council for Exceptional 

Children Special Conference on Instructional Technology commented on 

the function of formative evaluation: 

It is the formative evaluation process that results in 
specific revisions of a program to improve its rhetoric, 
instructional effectiveness, and acceptability. 2 

In Handbook on Formative and Summative Evaluation of Student 

Learning , Bloom, Hastings, and Madaus defined evaluation as: 

... the systematic collection of evidence to determine 
whether in fact certain changes are taking placvi in the 
learners as well as to determine the amount or degree of 
change in individual students. 3 

They distinguished between formative and summative evaluation on the 
basis of purpose, time at which evaluation occurs, and "... level 
of generalization sought by the items in the examination used to col- 
lect data Tor the evaluations"^ 



Ibid ., p. 59. 

2 

Lawrence Stolurow, "Instructional Technology" (paper pre- 
sented at a Council for Exceptional Children Special Conference on 
Instructional Technology, San Antonio, Texas, December 1-4, 1970), 
p. 75. 

3 

Benjamin S. Bloom, J. Thomas Hastings, and George F. Madaus, 
Handbook on Formative and Summative Evaluation of Student Learning 
(New York: McGraw-Hill Book Company, 1971), p. 8. 



^Ibid., p. 61. 



We have chosen the term "summative evaluation" to Indicate 
the type of evaluation used at the end of a term, course, or pro- 
gram for purposes of grading, certification, evaluation of pro- 
gress, or research on the effectiveness of a curriculum. . . . 

Formative evaluation is for us the use of systematic teaching 
and learning for the purpose of Improving any of these three 
processes. . . .1 

In the Preface to their book the authors explained that their inter- 
est is the Improvement of student learning, as the title of their 
book would indicate. 

Airasian also focused on formative evaluation for the improve- 
ment of student learning. He stated that formative evaluation 

" . . . seeks to Identify learning weaknesses prior to the completion 

2 

of instruction on a course segment. ..." He summarized differ- 
ences between formative and summative evaluation by indicating the 
verb tense used with each term: 

Formative evaluation provides data about how students are 
changing. Summative evaluation is concerned with how students 
have changed. . . .3 

In the report of his development of a model for formative 

evaluation Abador viewed formative evaluation as " . . . the process 

wherein developers of prototype Instructional systems collect and 



Ibid., p. 117. 



Peter W. Airasian, "Role of Evaluation in Mastery Learning." 
in Mastery Learning: Theory and Practice , ed. by B. S. Bloom 
(New York: Holt, Rinehart, and Winston, Inc., 1971), p. 79. 



^Ibid. , p. 78. 



analyze Information for purposes of correcting system deficiencies."' 
Abador had identified a formative evaluation component in the theo- 
retic models for the development of instructional systems he had 
examined, but he concluded that: 

. . c the formative evaluation procedures in such models are 
either too general for direct application to complex multi-media 
instructional treatments—or, If specific, provide techniques 
applicable to simple stimulus configurations such as textual pro- 
gramed instruction. . . . New tryout and revision procedures are 
needed to enable developers of prototype multi -media systems to 
more effectively identify problems, develop revision hypotheses, 
and design, integrate, and evaluate revisions. 2 



2, Models for Evaluation of 
Instructional Programs 

In the following discussion, five models for the evaluation of 
instructional programs will be described. The first four specify pro- 
cedures for the initial preparation of instructional programs. The 
fifth represents a model for summative evaluation conducted by an out- 
side evaluator. 



Model I .— Stake indicates that two main types of information are nec- 
essary for the evaluation of educational programs. The first typfi is 



Allan Je Abador, "Development and Validation of a Model for 
Formative Evaluation of Self-Instructional Multi-Media Learning Sys- 
tems" (paper presented at AREA Annual Meeting, April, 1972), p. 1. 

^Ibid., pp. 1-2. 
3 

Robert E. Stake, "Toward a Technology for the Evaluation of 
Educational Programs," in Perspectives of Curriculum Evaluation . AERA 
Monograph Series on Curriculum Evaluation, No. 1 (Chicago: Rand 
McNally and Company, 1967), p. 5. 



the Intents and outcomes, and the second Is personal judgments as to 
the quality and appropriateness of the Intents and outcomes. 

In another article Stake^ explained what his proposed evalua- 
tion program would Involve. Descriptions of what Intended anteced- 
ents or entry behaviors were expected, what Intended transactions or 
Instructional processes were planned, and what outcomes were antici- 
pated would be evaluated for their logical relationship to each other. 
Then the descriptions of what actually happened would be examined to 
determine If what was Intended actually occurred (see FIGURE 1). 
Finally, judgments of the value of the Instructional program would be 
made based on absolute standards reflected by the evaluator's per- 
sonal judgment and on relative standards reflected by comparison of 
the particular program to alternative programs. Program designers 
would prepare a rationale stating the basic purpose and philosophical 
background of their program which would assist the evaluators. 

Stake posed five questions which he felt should be answered 
prior to the initiation of evaluation procedures: 

1. Is this evaluation primarily descriptive, primarily judg- 
mental, or both descriptive and judgmental? 

2. Is this evaluation to emphasize the antecedent conditions, the 
transactions, or the outcomes alone, or a combination of 
these, or their functional contingencies? 

3. Is this evaluation to indicate the congruence between what is 
intended and what occurs? 

4. Is this evaluation to be undertaken within a single program or 
as a comparison between two or more curriculum programs? 



Robert E. Stake, "The Countenance of Educational Evaluation." 
Teachers College Record LVIII (April, 1967), pp. 527-38. 
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INTENDED ANTECEDENTS 



logical 
contingency 



INTENDED TRANSACTIONS 



logical 
contingency 



INTENDED OUTCOMES 



congruous 



congruous 



congruous 



OBSERVED ANTECEDENTS 



empirical 
contingency 



OBSERVED TRANSACTIONS 



empirical 
contingency 



OBSERVED OUTCOMES 



Figo 1. A representation of the processing of descriptive data„ 



I 



Robert E. Stake, "The Countenance of Educational Evaluation," 
Teachers College Record . April, 1967, p„ 533„ 



5. Is this evaluation intended more to further the development of 
curricula or to help choose among available curricula?l 

Stake here did not report the sequence in which the steps of 

his process would be followed, nor did he illustrate his process. No 

reports of projects in which his evaluation procedures had been used 

were located in the literature. 

Model II .— Briqqs in his monograph entitled Handbook of Procedures 
for the Development of Instructional Syst ems presented a model for the 
preparation of new instructional course material. His model, which 
encompasses course design, development, and evaluation, provides for 
the deliberate selection or creation of instructional materials on the 
basis of both learner characteristics and the nature of the compe- 
tencies which the course is supposed to develop, as well as on the 
basis of the characteristics of the material alternatives (see FIGURE 
2). The monograph is devoted to the design phase of Briggs' model. 
Briggs stated that formative evaluation procedures would start during 
the development and evaluation phases which he discussed briefly. He 
listed suggestions for formative evaluation which could be followed 
subsequent to what he called "formative design" steps taken during the 
development of first-draft materials in Steps 1-6. Briggs defined 
formative design as " . . . the use of performance tests (empirical 
data) for making the necessary decisions long before first-draft mate- 



hbid. , p. 539. 
2 

Leslie J. Briggs, Handbook of Procedures for the Design of 
Instructional Systems (Pittsburgh: American Institutes for Research, 
1970). 



12 



Q) 

E Q) 

^ E 

(/) O 

<o u 

4- C 1/1 
(U 

C (U u 







<u -o 




> c 




rO 








U (U 




Q) ^ 




•r-> 3 




^ 4^ 




O U 










U 


N -M 


c 


>> c/) 








«0 J- 


cr 


c o 






to 



CO 



t 




u 












€0 ^ 




(U O 












(U ^ 




$^ 4-> 






to 




Q) 


(U Q) 


> 


J- > 




a. o 



CM 



in 

> o 

•r- C 

U E 

a> J- - 
o in 

-Q -O 

o 

Q) to 
Q) Q.-0 
■M C 

CO <0 in 



















ro 4J 


ro 






Q-r— 






o 






o -o 


o. 






u 












■ 

3 






1 


to 


Q. 




O 


C 




<U WO 


(U o 






4-> 


u to 




in -M 


(U in 


c 




o; 


u n 


o -o 


ro O 










in 








c 




ro 




(U 




C > 


£ 


ro 


ro 


1— 4-> 




Q> Q> CT> 


ro 


o 


-o 




o «0 








ro 




to 








t 


9 




c 




o 


Q--0 




C 




Q) ro 


ro O 




•r- 3 


ro in "O 


Q-4-> 




Q) in 


E cn 




01 c 


a. 4-> 













<o to 



E 
O 

O in o 

J. 4-> I- 
tO 13 to 

in o 
ro > 

O 4-> 



cn 



Q--0 




3 C 


o 


O ro 


in 






C7) in 


o 


« +J 






in 


1— o 




to >>> 


E 




CO 4J 








00 












<*- 




ro 


in 






Q-T5 


ro 


O « 


•r— 


f— 4-> 






OJ 


> 


■M 




ro 


Q <*- 


E 












0 




in 


ro 


c 


•f— 


o 


■o 




OJ (U -M 


E ^ 












O 5 


u 




in 


1— -o 


Q) 


OJ c 


s. 


CO ro 




0 




ID 





CO 

o 

t—t 

CO 

> 

LU 



i 



4J 
U 




in 
■o 

o 



■o 
o 



4-> 
ro 



5 
O 



Osl 



c 

<0 

o 



tn 
+* 



u 

3 

in 



CI 
in 

0) 

o 

0) 
S. 

o 

s-j 

in 

0) 

s. 

3 

-o 

0) 

u 
o 
s. 



o 
o 

-o 
c 

rolp 

in 

o> • 

00 to 
0) 

9 cn 

O 01 

q: 

OJ 

•r- 

^ o 
inM- 

0) 

J m 
ro 01 
+* 

3 

in 



ERIC 



3 

4 



rials are ready for try-out." In his critique of his model written 
after he and his graduate studants had examined twenty other models 
for instruction drawn from military, industrial, educational, and 
governmental settings, Briggs observed several limitations in his 
model : 

The model is somewhat limited from the point of view of plan- 
ning the integration of materials, space, teachers, and learners 
into an administrative and management system for the operation of 
the learning environment. . . . 



Whereas the model may be Inadequate for skills of inquiry needed 
for advanced types of problem solving, it is clearly useful as a 
guide for planning instruction at many of the less advanced 
levels. 2 

Model II L — Baker and Schutz declared, "Most instruction is dispensed, 
3 

not developed." They characterized instructional development as 

" . . . essentially a cyclical process, ... a team effort, and . . . 
4 

user-oriented." They viewed an adequate instructional development 
program as one giving consideration to five program systems: Instruc- 
tional, Training, Installation, Accountability, and Modification. The 
Instructional system in the opinion of Baker and Schutz is the key 
system from which specifications for the other four systems are 
derived. All systems share common characteristics and are closely 
interdependent, although each system has a distinct function within 
the to^tal development program. 



'Ibid., p. 173. 

^Ibid., p. 185. 
3 

Robert L, Baker and Richard E. Schutz, eds.. Instructional 
Product Development (New York: Van Nostrand Reinhold Company, 1971), 
p. XV. 



Ibid . , pp. xv-svi. 
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Baker and Schutz listed seven components of their instruc- 
tional development cycle which cut across all five program systems 
and system characteristics. These components are: 

1. Formation 

2. Instructional Specification 

3. Item Tryout 

4. Product Development 

5. Product Tryout 

6. Product Revision , 

7. Operations Analysis 

For each of the seven components in the development cycle 
2 

Popham and Baker specified general rules (Appendix A). In addition, 
Popham described principles demonstrated to be effective in following 
the rules for activities within each component of the development 
cycle. These principles are: 

1. Provide relevant practice for the learner. 

2. Provide knowledge of results. 

3. Avoid the inclusion of irrelevancies. 

4. Make the material interesting. 3 

To produce the interest required in the last principle listed above, 
Popham urged the deliberate use of variety, humor, game-type situa- 
tions, suspense, and format variations. 



Ibid . , p. 131. 



W. James Popham and Eva L Baker, "Rules for the Development 
of Instructional Products," in I nstructional Product Development , ed. 
by Robert L. Baker and Richard t. Schutz (New York: Van Nostrand 
Reihnold Company, 1971). pp. 167-68. 



W. James Popham, "Preparing Instructional Products: Four 
Development Principles," in Instructional Product Development , ed. by 
Robert L. Baker and Richard E. Schutz (New York: Van Nostrand 
Reinhold Company, 1971), p. 171. 



Model IV .— Abador developed a model for formative evaluation which 
incorporated technical review, small group tryout with tutorial 
assistance by the prototype designer, and large group tryout. He sub- 
mitted his initial model (see FIGURE 3) to seven faculty members who 
had developed self-instructional multi-media lessons. He found that 
the model was unacceptable to the faculty members because of the time 
Involved in carrying out the steps, the high cost of nf educing revi- 
sions, and the difficulties of coordinating prototype production* 
large group tryouts, revisions, and course schedules. In addition, 
most of those he asked considered themselves to be experts in their 
particular subject matter and ways to present it and felt a technical 
review was unnecessary.^ 

Abador developed another model for formative evaluation in 
which he substituted a group tryout and debriefing procedure for the 
tutorial and large group components of the initial model (see FIGURE 
4). The revised model contained the technical review step to avoid 
errors or critical omissions of content. 

To test the validity and effectiveness of his revised model, 
Abador had three Michigan State University faculty members use the 
model to prepare prototype lessons. Students were randomly assigned 
to control groups and experimental groups to take the lessons and 
were stratified on the basis of their scores on the Scholastic Apti- 
tude Test. After the control group students had each completed the 
prototype lessons, they met as a gro-p with the lesson developer in a 



Abador, "Development ^,nd 'i<. it"' r ," pp. 3-4. 
Ibid . , p. 6. 
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debriefing session to discuss instructional deficiencies in the les- 
sons and to identify feasiLle solutions to correct those deficiencies. 
Revisions were made in the lessons before students in the experimental 
groups were permitted to begin instruction. 

Abador reported that when group mean achievement, gain scores, 
percentage of students achieving mastery (defined as eighty percent 
or more correct on the post-test), and student attitude were con- 
sidered, significant differences favoring the revised lessons were 
found for all four variables on two experiments, m significant dif- 
ference on the post-test measure favoring the revised lessons was 
found on a third experiment. Abador emphasized that debriefing ses- 
sions with the control group enabled the lesson developers to identify 
precisely where weaknesses existed in the lessons and to collect sug- 
gestions regarding how those deficiencies might be remediated. He 
observed that the face-to-face debriefing sessions between students 
and lesson developers can be devastating and require a strong com- 
mitment to the principles of tryout and revision. 

Model v .— Glass^ applied a prototype evaluation format to the 
appraisal of an educational product already on the market, an 
instructional 100-foot cassette tape recording of a presentation 
entitled "Evaluation Skills" given by Dr. Michael Scriven. The 
model covered the following items: 

1 . Product description 

2. Goals evaluation 



Gene V. Glass, "Educational Product Evaluation: A Prototype 
Format Applied," Educational Researcher . I (January, 1972), 
pp. 7-10, 16. 



3. Clarification of point of entry of the evaluator 

a. Irreversible decisions 

b. Reversible decisions (Enter the evaluator) 

4. Trade-offs 

5. Comparative cost analysis 

6. Intrinsic (secondary) evaluation 

a. Technical quality 

b. Content evaluation 

c. Utilization of uniqueness of medium 

d. Survey of availability 

7. Outcome (primary) evaluation 

8. Summatlve judgments and recommendations 

9. Circumstances modifying the summatlve judgments (scope and 
value claims) 

Glass's prototype model was prepared for the outside evalua- 
tor to follow In appraising a finished Instructional product. 

3. Summary 

Authorities have distinguished between formative and summatlve 
evaluation and have developed models for authors of Instructional pro- 
grams to follow. Stake's plan for evaluation provides a ijeneral out- 
line for Instructional development projects which make evaluation an 
Integral part of the project. Briggs' model places emphasis on the 
selection of available materials or the design of new materials In 
order for students to reach Instructional objectives. Abador's model 
offers a compromise acceptable to experienced developers of multi- 
media self-Instructional lessons and requires the capability of 



assembling tryout students for a final debriefing session with the 
lesson developer. Glass reports the results of his having used a 
model appropriate for the evaluation of finished products. Baker and 
Schutz outline a practical program for the newcomer in program devel- 
opment to follow. 

Factors influencing the choice of a particular model would 
seem to include the purpose for and the scope of the evaluation, the 
point at which the evaluation is to be initiated, and the person to 
whom the task of evaluation is assigned. 

B. Computer-Assisted Instruction 
1. Definitions 

Computer-assisted instruction (CAI) defined broadly includes 
" . . . the entire field of computer uses for instruction in which 
there is an interaction between student and machine."^ A more narrow 
definition limits CAI to " . . . tutorial exercises of computerized 

2 

programed instruction; ..." 

In a paper presented at the Conference on Applications of 
Digital Computers to Automated Instruction held in 1963, Rigney dis- 
cussed the computer as a teaching machine with an " . . . interactive 



Karl L. Zinn and Susan McClintock, A Guide to the Literature 

on Interactive Use of Computers for Instruction (2nd ed.; Stanford: 

ERIC Clearinghouse on Educational Media and Technology, 1970), p. 22. 

^Ibid. 



capacity in relation to Individual differences In learning." He 
identified two ways the interactive capacity could be applied in stu- 
dent computer interactions. In the first, the computer would domi- 
nate by controlling the interactions according to rules specified by 
the author. In the second, the student would control the interaction 
by managing the general rules and determining what to ask the com- 
puter. Rigney described three types of learning environments in which 
the computer had been or could possibly be used: 

1. In team training in games such as war gaming or tactical situ- 
ations with the computer simulating environmental events. 

2. In advanced technical and professional training of an indi- 
vidual with the computer serving as an " . . . inanimate 

assistant performing a variety of routine tasks for him, 

2 

including pure information retrieval functions." 

3. In individual verbal tasks using teaching machine programs 

under computer control. 

3 

Atkinson and Wilson in 1968 mentioned in their sampling of 
computer applications in education the use of the computer for 



Joseph W. Rigney, "Potential Uses of Computers and Teaching 
Machines," in Programmed Learning and Computer-Based Instruction , ed. 
by John E. Coulson (New York: John Wiley and Sons, Inc.,, 1961), 
p. 156. 

^Ibid. 

'Richard C. Atkinson and H. A. Wilson, eds., "Computer-Assisted 
Instruction," in Computer-Assisted Instruction: A Book of Readings , 
(New York: Academic Press, 1969), pp. 8-9. 



drill-and-practice procedures, tutorial programs, simulations and 

games. They also discussed the computer as a tool for computation in 

math and science. 

Holtzman^ specified CAI as just one of several applications 

of the computer in education. He described CAI as a system in which 

the student at a terminal interacts directly with the instructional 

materials which are stored in the computer or at the terminal. The 

materials can range from dril 1-and-practice exercises and tutorial 

interactions to complex sin.ulations and problem-solving. 

2 

Zinn and McClintock have identified five categories of cur- 
rent computer use in education. These categories are: 

1. Drill, author-controlled tutorial and dialogue tutorial .— In 
these kinds of programs the author prepares the objectives 
and describes the course content in detail. The drill stra- 
tegies are carried out according to the author's specifica- 
tions. The tutorial programs allow for increasing amounts of 
variation in student response and initiative, but control of 
the interaction remains with the author. 

2. Simulation and gaming . — In these applications the author out- 
lines a general program model with appropriate replies for 
whatever responses a student supplies in answer to questions. 
The author does not specify frame-by-frame program content as 
he does in the drill and tutorial programs. 



Wayne H. Holtzman, "Computers in Education," in Computer- 
Assisted Instruction, Testing, and Guidance , (New York: Harper and 
Row, Publishers, 1970), pp. 5-8. 

2 

Zinn and McClintock, Guide to the Literature , pp. 6-8. 
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3. Information handling, computation, and display . —Essentlanv in 
these applications the computer serves as an information proc- 
essing device and intellectual aid. 

4. Tool for author and researcher . --Some computer systems have 
capabilities for interactive composition and revision of mate- 
rials. They may also generate additional material for indi- 
vidual students from rules provided by the course author and 
may be programed to assist with data analysis and with 
reaching decisions in both research and modification of 
instructional systems. 

5. I nstructional management . --The computer takes over clerical 
work and routine handling of performance records to aid 
instructional management. 

Holtzman^ in his discussion further described computer-managed 
instruction as a "support" system for the teacher to use in pre- 
scribing, storing, and retrieving diagnostic, achievement, and other 
reference information about individual students. He distinguished 
instructional management from educational management in which the com- 
puter also serves as a support system. In the latter application the 
computer handles information regarding admission of students, regis- 
tering, updating records, scheduling, and reporting grades. 

Focusing on the use of computer programs to individualize 
instruction, Suppes classified three possible levels of interaction 
between the student and computer program. At the simplest level are 



1 



Holtzman, "Computers in Education," p. 6. 
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individualized drill-and-practice systems "... which are meant to 

supplement the regular curriculum taught by the teacher."^ At the next 

and deeper level are tutorial systems "... which take over the 

responsibility both for presenting a concept and for developing skill 
2 

in its use. Suppes considered the deepest level of interaction to be 

the dialogue system "... aimed at permitting the student to conduct 

3 

a genuine dialogue with the computer." Writing in 1968, he made the 

following prediction: 

I would predict that within the next decade many children will 
use individualized drill-and-practice systems in elementary school; 
and by the time they reach high school, tutorial systems will be 
available on a broad basis. Their children may use dialogue sys- 
tems throughout their school experience. 4 

5 

Hall summarized the characteristics of present computer appli- 
cations to instruction; 

1. Ability of a pre-stored program in a computer system to evalu- 
ate a student's response and provide information regarding the 
accuracy of the response. 

2. Active responding by the student. 

3. Ability to individualize instruction not only at the level of 
achievement but also in reference to specific interests and 
abilities of the student taking the course. 



Patrick Suppes, "Computer Technology and the Structure of 
Education," in Computer-Assisted Instruction: A Book of Readings , ed. 
by Richard C. Atkinson and H. A. Uilson (New York: Academic Press, 
1969), p. 43. 

^Ibid . ^jbid., p. 44. ^ Ibid . 



Keith A. Hall, "Computer-Assisted Instruction: Problems and 
Performances," Phi Delta Kappan , June, 1971, p. 628. 
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Feldhusen has stated: 

There appear to be some things which CAI will become able to do 
better than any other media: 

1. Secure, store and process Information about the student's per- 
formance prior to and/or during Instruction to determine sub- 
sequent activities In the learning situation, 

2. Store large amounts of Information and make It available to the 
learner more rapidly than any other medium, 

3. Provide programed control of several media such as films, 
slides, TV, and demonstration equipment, 

4. Give the author or teacher an extremely convenient technique 
for designing and developing a course of instruction, and 

5. Provide a dynamic interaction between student and instructional 
program not possible with most other media. 1 

The dynamic interaction capability characteristic of current 
CAI systems contributes both to the usefulness of CAI systems in educa- 
tion and to the complexity of CAI program development and evaluation. 
The investigation of an effective approach to the preparation of CAI 
course material for use in current dynamic interactive CAI systems 
would s.-m to be worth careful consideration. 

2. CAI Course Design 

Gage commented in 1963, "The heart of machine teaching is the 

2 

program, ..." This would seem to be particularly true for CAI. 



John H. Feldhusen, A Position Paper on CAI Research and Devel- 
opment , (Stanford: ERIC Clearinghouse on Educational Media and Tech- 
nology, 1970), pp. 1-2. 



N. L. Gage, "Paradigms for Research on Teaching," in Handbook 
of Research on Teaching , (Chicago: Rand McNally and Company, 1963) , 
p. 132. 



26 

Hlckey^ discussed three of the decisions CAI course authors 
must make during the preparation of their programs: what order con- 
cepts are to be presented, whether they are to be presented as rules or 

as examples or both, and in what sequence student responses are to be 
2 

reinforced. Dean advised course authors to learn what the capabil- 
ities for interaction were in the particular CAI system for which they 

3 

anticipated writing course material. Pipe expressed concern over stu- 
dent attitudes and urged authors to talk to members of the population 
they intended to instruct before attempting to write any course 
material . 

4 

Zinn from his work concluded that available texts, students' 
answers to constructed response questions, and transcriptions of stu- 
dent/instructor interactions could be used as sources of first-draft 

CAI course material . 

5 

Wassertheil found that writing concept units rather than 
larger units, a procedure she called a modular strategy, had certain 



Albert E. Hickey, ed., Computer-Assisted Instruction: A Sur- 
vey of the Literature (3rd ed.; Newberryport, Mass. : Entelek Incor- 
porated, 1968), p. 79. 

2 

Peter Dean, "Course Authoring Techniques for CAI," In Adult 
Basic Education: An Institute for State and University Level Adult 
Basic Education Personnel , ed. by Mary Louise Ceilings (Ra1eiqh» N. C. : 
Adult Learning Center, 1971), p. 118. 

3 

Peter Pipe, "Student Attitudes - The Neglected Dimension," 
NSPI Journal , IV (September, 1965), p. 13. 

4 

Karl L, Zinn, "Computer Technology for Teaching and Research 
in Instruction," Review of Educational Research , XXXVII (December, 
1968), p. 19, ^ 

5 

Sylvia Wassertheil, "A Modular Strategy for Developing 
Computer-Assisted Instruction Programs," NSPI Journal , VII (May, 1968), 
pp. 618-34. 



advantages. By concept units she meant materials designed to teach a 
particular concept, skill, technique, or set of facts. Among the 
advantages she attributed to writing concept units were greater flexi- 
bility in arrangement of units, greater chance of success in writing 
units, and less chance of traumatic revisions. 

Meredith and Ferguson^ described a method for measuring the 
value of student feedback in aiding refinement of CAI frames presenting 
open-ended questions for which there might be several legal or accept- 
able responses. They collected fifty responses to a question, checked 
the students' answers and the appropriateness of the proposed frame 
replies to the students' answers, and then revised the frame replies. 
They found that by using this procedure they could construct more 
appropriate intercepts or replies to anticipated correct and incorrect 

student answers. 

2 

Wachner encouraged authors to be creative in their programs, 
to adopt attention-getting techniques similar to those used on televi- 
sion, and to consider both horizontal and vertical continuity. Hori- 
zontal continuity covered the interrelationship of media in use at the 
same time, while vertical continuity referred to the sequencing of 
media. 



Joseph C. Meredith and Douglas Ferguson, "Student Feedback As 
a Tool in CAI Frame Development," International Journal of Experimental 
Research in -Jucation , VII (1970), pp. 221-302. 



Barbara Wachner, "I Forgot How to be Interesting," NSPI 
Journal , VI (March, 1967), pp. 10-14. 
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Scott and Yelon have indicated that other persons can be help- 
ful 1n determining program effectiveness during the developmental 
stages. They have suggested that modifications in course design can be 
made on the basis of reactions of individual students or small groups 
of students prior to any field testing. The students must, however, 
understand that the materials, not they, are being tested. First 
bright students, who possibly would be less confused by instructional 
sequences in rough form, and then slower students can give some indica- 
tion of how appropriate and effective the materials are. Scott and 
Yelon have also pointed out that "subject matter sophisticates" (SMS's) 
can be helpful to authors in identifying inconsistencies and format 
errors and can evaluate the content of the course. As to how much try- 
out with individual students or SMS's is enough, Scott and Yelon have 
stated: 

A rule of thumb for determining a cut-off point is to stop when two 
or three successive sessions have shown that target population stu- 
dents can perform the instrjctional outcomes wUhout help frc**; the 
person conducting the try-out. ^ ^. 

C. Evaluation of Computer-Assisted Instruction 
Course Material During Initial Preparation 

2 

Rogers'* several years ago reviewed problems in CAI and observed 

that lack of quality CAI course material constituted a major problem* 
He called attention to the need for evaluated course material So 



Roger 0. Scott and Stephen L. Yelon, "The Student as a Co- 
Author - The First Step In Formative Evaluation," Educational Tech- 
nology , October, 1969, p. 76. 

2 

James L. Rogers, "Current Problems in CAI," Datamation , 
(September, 1968), pp. 28-33. 
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Cartwright has iridntified recent trends in curriculum evalua- 
tion: evaluation is b-jcoming acceptable and broader in base; as the 
contribution that fornative evaluation can make to curriculum develop- 
ment rev-:eives greater recogni:ion, there is a corresponding decrease in 
emphasis on summative evaluation; and, in spite of this recognition, 
" . • . the large majority of CAI publications and papers that have 
become available in the last two years still are reporting summative 
evaluation activities. . , . 

1. Formative and Sunmative Evaluation 
Cartwright and Mitzel described both the formative and summa- 
tive procedures they followed during the preparation of a three-credit 
CAI course designed for regular classroom teachers primarily in rural 

i 

areas entitled "Early Identification of Handicapped Children," During 

the formative evaluation procedures, which covered approximately six 

months, fifteen students took the course while a proctor observed and 

recorded any student comments and program bugs. Technical problems 

went to the programer and content problems were given to the author i 

who made necessary changes. Once all revisions has been organized, the J 

I 

course was revised and a second pilot group of fifteen students took | 

the course unattended by a proctor. In addition, two graduate students J 

I 

— r~ ^ 

'cartwright, "Issues," p. 2. | 

2 f 
6. Phillip Cartwright and Harold Mitzel, Development of a CAI | 

Course in the Identification and Diagnosis of Handicapping Conditions r 

in Children; Final Report No. R-44 (University Park. Pa.': tKp } 

Pennsylvania State University CAI Laboratory, June, 1971). ' 
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1n special education completed the course and submitted their evalua- 
tion reports. Finally, 115 inservice teachers completed the course. 
Extensive revisions were made as a result of the analyses of the 
responses, requests for assistance, and response latencies collected 
from these students. 

To conduct a sumn^ative evaluation of the course, on-campus stu- 
dents who registered for "Introduction to the Education of Exceptional 
Children" or CARE 1^ were randomly assigned to conventional instruction 
(CI) and to CAI. Objectives for both courses were the same; in fact, 
the teacher of the CI class had been one of the CAI course authors. 
Using time to complete the course and score on the 75-1tem final exam 
as variables, the authors reported that analyses of their data indi- 
cated the CAI students {n=27) scored signif -^cantly higher than CI stu- 
dents (n=87) on the final exam and completed the course in twelve hours 
less time than the CI students. 

At the University of Texas CaI Laboratory a summative evalua- 
tion of CARE 1 was conducted to determine both the acceptability of 
CARE 1 to inservice teachers in Texas and the value of CARE 1 for pre- 
service teacher training at the University of Texas^ Thirteen graduate 
students in special education examined the content and presentation of 
CARE 1 and estimated the course to be feasible for both inservice and 
preservice training. 



'Computer Assisted Remedial Education 1 or CARE 1 is the three 
credit course entitled "Introduction to the Education of Exceptional 
Children." 

2 

Agnes Edwards and Wilson A. Judd, Evaluation of CARE 1 for Pro- 
jected Use at The University of Texas: Technical Report No> 12 (Austin, 
Texas: The University of Texas at Austin Computer Assisted Instruction 
Laboratory, January, 1972). 



During a sunnier mservics teacher training workshop, seventeen 
teachers from El Paso completed CARE 1 at the CAI Lab. In addition to 
their on-line time, these teachers met for a one-hour discussion each 
day to consider the course content in light of each teacher's own 
teaching experience. At the completion of the final exam, the teachers 
were asked to fill out a forty-two item Student Opinion Survey dealing 
with such things as attitudes toward CAI, likes and dislikes of the 
course, and operation of the equipment. The Lab reported that the 
teachers expressed "a relatively favorable opinion."^ 

During the 1971 fall term, selected chapters of CARE 1 were 
used as an adjunct to a regular course entitled "Behavioral Science 
Foundations of Elementary Education." The purpose was to detennine the 
efficacy of CAI compared to conventional instruction and to see how 
well CAI worked in combination with conventional instruction. The 
sixty-nine students who had registered for the conventional courses 
were randomly assigned to one of three groups. All three groups 
attended class lectures. Only groups A and B were provided with an 
abridged version of the CARE 1 Handbook . Groups B and C met weekly for 
a one-hour discussion session. Group A worked through the on-line 
CARE 1 course material. Groups B and C were not exposed to any CAI 
material. Analysis of pre- and post-test adjusted means for the three 
groups showed a significant difference favoring Group A on both the 
CARE 1 section of the test and on the total test. There were no sig- 
nificant differences among the three groups on the lecture portions of 



Ibid ., p. 13. 



the test, but the differences that did occur were in favor of Groups A 
and B, the groups that used the Handbook . From the results of this 
summative evaluation, the staff at The University of Texas concluded 
that: 

Although CARE 1 was designed primarily for the training of inserv- 
ice teachers, this evaluation shows that the course is also appli- 
cable to preservice training, and can be used as an adjunct to an 
existing conventional courseJ 

2 

Confer reported another summative evaluation of a CAI course 
designed to teach general math. Students, all repeaters in general 
math, were randomly assigned to regular class instruction and to CAI 
instruction during a summer school session. Performance at the end of 
instruction in computation and problem-solving was measured with the 
Stanford Achievement Test (SAT). Analysis of covariance indicated no 
significant differences between the two groups on SAT -cores. Corf^** 
concluded that his results neither confirmed nor rejected CAI as a 
method of instruction. Among his recommendations was the need for an 
analysis of all students' responses to help determine necessary changes 
in the CAI general math course. 

In a speech at the Association for the Development of Instruc- 
tional Systems in 1971, Cartwright stated: 

It is unlikely that summative evaluation per se will improve 
the quality of instruction. Formative evaluation, however, is a 
model that can be used to improve the quality of instruction. 



' Ibid ., p. 22. 

^Ronald W. Confer, "The Effect of One Style of CaI on the 
Achievement of Students Who Are Repeating General Math," (unpublished 
Ph.D. dissertation, University of Pittsburgh, 1970). 



It seems to me that criterion-referenced instruction as a goal 
and formative evaluation as a method is the way to go at this point 
in time in the development of CAI J 



2. Criteria for Evaluation of CAI 
Seltzer has written: 

What the computer can and cannot do is a matter of research and 
fact. What the computer should and should not do in instruction is 
based on value judgments. . . .2 

Seltzer suggested that, in order to be in a position to make value judg- 
ments, criterii^n statements should be drawn up for use in evaluating 
the selection of the computer to assist in any particular instructional 
process. The criterion statements proposed are: 

1. If the computer poses a unique solution to an important problem 
in the instructional process, then it should be used regardless 
of the cost involved, 

2. If the computer is more efficient or effective and the cost of 
its use to instruct is minimal, then it should be used. And 
conversely, 

3. If the cost of development and use of the computer in instruc- 
tion is relatively high with the relative efficiency or effec- 
tiveness only marginal, then the computer should not be used in 
the instructional process. 3 

These criterion statements look at CAI cost, effectiveness, and 
efficiency in comparison to alternative means of instruction^ They 



^Cartwright, '^Issues," p. 9. 
2 

Robert A. Seltzer, "Computer- Assisted Instruction - What It 
Can and Cannot Do," American Psychologist , XXVI, No. 4 (April, 1971), 
p. 373. 

^Ibid., p. 375. 



could logically be considered during the initial design of a proposed 
CAI appilcatloi. to instruction before much instructional material had 
been developed. 



3. Model for CAI Course Preparation 
and Evaluation 

Bunderson constructed a prescriptive model for the design of 
CAI course material. He explained the circumstances which prompted his 
effort: 

The Instructional design model described in this chapter was origi- 
nated to provide management and quality control for curriculum 
development, to provide a bridge between the curriculum development 
and basic research activities of the laboratory, and to serve as a 
focus for teaching students and others how to design quality CAI 
programs. Its development was Influenced by the author's attempts 
to adjust to a joint appointment between educational psychology and 
computer science and to communicate with staff members and students 
from both fields.' 

Bunderson discussed the activities to be performed by the 
instructional designer, their approximate sequence, and the product of 
each activity. The design activitie:> in the sequence Bunderson out- 
lined are: 

1. Intent and justification 

a. Write societal needs. 

b. Write institutional needs, 

c. Write program goals. 

1. Describe job requirements. 

2. Describe student population. 

3. Describe institutional constraints, 

d. Write justification for CAL 



C. Victor Bunderson, "The Computer and Instructional Design," 
in Computer-Assisted Instruction, Testing, and Guidance ^ ed. by Wayne 
H. Holtzman (New York: Harper and Row, Publishers, 1970), p. 46. 
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2. Instructional design: analysis 

a. Derive operational requirements from goals. 

1. Derive terminal objectives. 

2. Set entering performance standards. 

3. Consider effect of constraints on program design. 

b. Behavioral analysis, 

1. Obtain intermediate objectives through analysis of 
terminal objectives. 

2. Construct learning hierarchy. 

c. Analysis of learner traits. 

3. Instructional design: synthesis 

a. Specify interface. 

1. Display and response devices. 

2. Representation. 

b. Construct individualizing flow chart. 

1. Hierarchy-based gating mechanisms. 

2. Trait-by-treatment branches. 

3. Continuously adaptive mechanisms. 

c. Write working draft. 

1. Construct curriculum-embedded tests controlling major 
flow. 

2. Write steps and describe format of steps. 

4. Produce program materials. 

a. Code from author's draft- 

b. Produce media. 

c. Debug code and proof media. 

5. Evaluate and revise, 

a. Editorial evaluation- 

b. Internal empirical evaluation. 

c. External empirical evaluation. 
1 Validation testing. 

2. Longitudinal validation. 

6. Use of feedback 

Return to any previous step as indicated by evaluation, revise, 
and recycle. 

In his discussion of parts of his model, Bunderson observed 
that the construction of a learning hierarchy (see above, 2.b.2.) 
seems 

. . . readily applicable to any cumulative subject matter such as 
mathematics, much of science, anc! even music. It seems less appli- 
cable to highly verbal areas. 1 



4. Summary 

A survey of the literature related to both evaluation and CAI 
reveals that models for formative evaluation are available for use in .« 
developing course materials, that authorities urge formative evaluation 
be incorporated into initial CAI course development projects, and that 
to date formative evaluation procedures have not been reported in many 
completed projects In the one model designed specifically for CAI 
course preparation, the author observed that some of the activities he 
outlined were more suitable for subject matter with Inherent structure 
rather than for highly verbal subject matter on which several struc- 
tures might be imposed. 

Authorities continue to stress the need for formative evalua- 
tion during initial CAI course preparation for purposes of course 
improvement. They see little information in summative evaluation 
results that can help authors locate course weaknesses or errors. 

The application by a course author of a formative evaluation 
model to the initial preparation of a CAI course would seem strongly 
indicated. 



CHAPTER 111 



PROCEDURES 

In order to complete this study involving the use of formative 
evaluation during the initial preparation of a CAI course, the in ^^sti- 
gator performed the following activities: 

1. Reviewed professional literature pertaining to evaluation of 
instructional programs, computer-assisted instruction, and 
evaluation of CAI course material during initial preparation 
(see Chapter II}. 

2. Selected a model for formative evaluation appropriate for an 
author to use during the preparation of a CAI course (see 
Chapter IILB), 

3. Conducted a formative evaluation during the preparation of a 
CAI course (see Chapter IV, A-G). 

4- Identified information suitable for possible inclusion In a 
manual for use by other authors conducting formative evaluation 
during the preparation of instructional programs (see Chapter 
IV, H). 

A. Definition of Terms 
Computer-Assisted Instruction » —Computer-assisted instruction or CAI is 
a means of individualizing instruction for students through the use of 
student/computer interactions. For purposes of this study CAI 
referred to the IBM 1500 Instructional System with capabilities for 
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tutorial and simulation programing strategies With this system course 
materials can be presented to students on a cathode ray tube (CRT) 
which resembles a television screen, on pre-recorded tapes, and on 16mm 
color or black and white microfilm projected from an image reel. The 
system can accept and process student responses entered either on the 
typewriter keyboard or with a light pen and can consult decision rules 
specified in advance by the course author to direct each student's path 
and progress through the course. The IBM 1500 Instructional Systems 
used in this study were located in the Computer Assisted Instruction 
Laboratory at The Pennsylvania State University and n The Pennsylvania 
State University CAI Mobile Laboratory. The Mobile Laboratory, a com- 
plete fifteen-terminal CAI system, is housed in a forty-foot long 
expandable van which can be transported from one location to another by 
diesel tractor. (See Appendix B for a more complete description of the 
IBM 1500 Instructional System and for a picture of the Mobile 
Laboratory. ) 

CAI Course , —For purposes of this study the CAI course was "Education 
of Visually Handicapped Children"^ designed for regular classroom 
teachers in rural areas. This course will be called CARE 4 in the fol- 
lowing discussion. (See Appendix C for a list of course objectives and 
a description of the course content and Appendix D for a list of refer- 
ences used in writing the course.) 



The Special Project funded to develop "Education of Visually 
Handicapped Children" was performed pursuant to a grant under Title VI 
of the Education of the Handicapped Act (PL 91-230), Special Project 
Grant OEG-0-71 -1604) at the University of Pittsburgh in the Department 
of Special Education and Rehabilitation in cooperation with the CAI 
Laboratory at The Pennsylvania State University. 
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Formative Evaluation .— For purposes of this study formative evaluation 
was "... outcome evaluation of an intermediate stage"^ in the devel- 
opment of CARE 4 for the purpose of determining deficiencies and 
strengths during initial preparation. 

OfMJjie. --"performed outside of the operation of the central processor 

2 

of a computing system" 

Off -Line Materials . --materials in hard copy or in some form accessible 
away from the CAI student terminal 

On-Line c— "connected directly to the central computer" 

On-Line Material .—course material available to student only via the 
CAI system at a student terminal 

On-Line Time .— time spent working at a CAI student station in inter- 
action with the computer 

Semi -structured I ntervi ew . — i ntervi ew conducted while following an 
interview schedule only as a guide to discussion, not necessarily 
adhering to the exact wording of the questions or their order of 
written presentation 



Michael Scriven, "The Methodology of Evaluation," Perspectives 
of Curriculum Evaluation . AERA Monograph Series on Curriculum Evalua- 
tlon. No. 1 (Chicago: Rand McNally and Company, 1967), p. 51 • 

2 

Karl L. Zinn and Susan McClintock, A Guide to the Literature 
on Interactive Use of Computers for Instruction (2nd ed.: Stanford: 
ERIC Clearinghouse on Educational Media and Technology, 1970), p. 24. 



Ibid. 
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B. Selection of a Model 
A review of the literature related to formative evaluation Indi- 
cated that the selection of a suitable model for formative evaluation 
would seem to depend upon the person made responsible for conducting 
the formative evaluation, the point at which evaluation would begin, 
and the purpose of the evaluation. For purposes of this study, it was 
determined that the formative evaluation model selected would have to 
be suitable for a CAI course author, the investigator, to use from the 
early stages of course design to the completion of the course develop- 
ment for purposes of course Improvement. 

Five models for formative evaluation were examined: 
Model I Stake's theoretic model for curriculum evaluation 
(above, p, 8) 

Model II Briggs's procedures for designing multi-media instruc- 
tion (above, p, 11) 

Model III Baker and Schutz's cycle for instructional product 
development (above, p* 13) 

Model IV Abador's model for formative evaluation of self- 

I 

instructional multi-media learning systems (above, 1 

(p. 15) I 

i 

Model V Glass's prototype model for the outside evaluator of i 
instructional programs (above, p. 18) w 

i 

Bunderson's prescriptive model, which is specifically for the develop- j 

• -4 

ment of CAI course materials, was also studied (above, p, 34), I 

Stake's model includes both an outside evaluation of the goals, | 

rationale, and course material and a comparison of the intended results | 
of the instruction with the actual results. Stake did not, however. 
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provide the potential user with any Illustration of how his model might 
be followed or the sequence In which the steps of his model should be 
carried out. No reports of anyone having used his model were located 
In the literature examined* 

Briggs's model specified procedures to assist In the selection 
of materials for Instructional programs, but the emphasis of the model 
falls on the design phase of course preparation. Briggs has stated 
that his model may be inadequate for skills of Inquiry necessary for 
higher levels of learning and problem-solving.^ 

Abador's model requires that tryout students be available for a 
final group debriefing session with the Instructional program devel- 
oper, a procedure which could not easily be followed given the target 
population and field testing possibilities for the CARE 4 project. The 
model provides for no assessment of the overall project rationale and 
objectives. 

Glass's model outlines procedures for the outside evaluator to 
follow in looking at a finished instructional product; it is not suit- 

k 

able for use by the developer of the product during initial preparation. '| 
Bunderson's model does include essentially the same steps as S 

the Baker and Schutz cycle; but, according to Bunderson,^ the proce- ^- 

f 

dures are applicable to highly structured subject matter. During the ^? 

i 

initial design stage, the content proposed for CARE 4 was not found to ? 
be highly structured or hierarchical in nature; on the contrary, one of | 

I 

Leslie J. Briqgs, Handbook of Procedures for the Design of f 
Instructional Systems (Pittsburgh, Pa.: American Institutes for 
Research, 1970), p. 185. i 

2 ' 
C. Victor Bunderson, "The Computer and Instructional Design," 
in Computer-Assisted Instruction, Testing, and Guidance > ed. by Wayne 
H. Holtzman (New York: Harper and Row, Publishers, 1970), p. 56. i 
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the early tasks the authors found necessary was the imposition of a 
structure on the proposed content in order to bring about efficient 
learning. ^ 

The programed-style book presenting the Baker and Schutz cycle 

for Instructional program development and the principles for using it 

has been described as "Wide in scope with sufficient depth to make this 

2 

a must on the novice's bookshelf." The instructional development 
cycle is basic to establishing the requirements for the components of a 
larger evaluation system which can include training to equip personnel, 
installation of the instructional product, accountability and proce- 
dural adequacy of the operating program, and a procedure for modifying 
all programs in the system. The Baker and Schutz model does not 
specify an outside evaluation such as Stake has suggested.^ 



Glaser and Resnick have suggested that the logical order of a 
body of information is not necessarily the order most appropriate to 
bring about learning in students who are not already familiar with that 
body of information. See Robert Glaser and Lauren B. Resnick, Instruc- 
tional Psychology (Pittsburgh, Pa.: Learning Research and Development 
Center, University of Pittsburgh, 1972), p. 209. 



Paul A. Twelker, Floyd D. Urbach, and James E. Buck, The Sys- 
tematic Development of Instruction: An Overview and Basic Guide to the 
Literature (CorvallisJ Oregon: United States International University 
in Oregon, March, 1972), p. 15. 



Robert '.. Baker and Richard E. Schutz, eds.. Instructional 
Product Development (New York: Van Nostrand Reinhold Company, 1971), 
pp. xvi-xviii. 



Robert E. Stake, "Toward a Technology for the Evaluation of 
Educational Programs," in Perspectives of Curriculum Evaluation . AERA 
Monograph Series of Curriculum Evaluation, No. 1 (Chicago: Rand 
McNally and Company, 1967), p. 5. 
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The investigator, after consultation with members of the disser- 
tation committee, elected to follow the Baker and Schut? cycle for 
instructional product development but with the addition of an outside 
evaluation step to the Instructional Specification stage of the cycle. 
The revised model for formative evaluation which the investigator fol- 
lowed called for these procedures which will be discussed in detail in 
the next section: 

1 . Formulation 

2. Instructional Specifications and Outside Evaluation 

3. Item Tryout 

4. Product Development 

5. Product Tryout 

6. Product Revision 

7. Operations Analysis 

Activities 

Although the procedures carried out in this study are listed 
sequentially below, many were actually carried out concurrently- They 
took place between June, 1971 and August, 1972. 

1 - Formulation 

During the Formulation period the investigator wrote a rationale 
for a CAI course about the education of visually handicapped children 
for regular classroom teachers. The investigator drew upon information 
available from The Pennsylvania State University CAI Lab regarding one 
solution to problems of inservice training of teachers in rural areas. 
The solution was delivery of courses to rural arpa teachers via a 
mobile CAI laboratory. (See Appendix E for a copy of the rationale.) 



As a part of initial course design, the investigator surveyed 
the existing professional literature regarding the education of visu- 
ally handicapped children. (See Appendix D for a list of references 
used in writing CARE 4 course material.) 

In addition to examining the professional literature, the 
investigator conducted semi -structured interviews with three regular 
classroom elementary teachers and one teacher of physically handi- 
capped children. These teachers were located through the state con- 
sultant and the coordinator and itinerant teachers in a program for 
visually handicapped children operated by an intermediate unit near 
the University of Pittsburgh. The total number of teachers inter- 
viewed was kept small because of the constraints on conducting inter- 
views imposed by the U. S. Office of Education which was providing 
funds for the development of CARE 4. Each of the four teachers 
interviewed had had at least one visually handicapped child in her 
class. The questions asked (Appendix F) were designed to determine 
what information these teachers thought, based on their own experiences 
with visually handicapped children, would be helpful to other regular 
classroom teachers who would have visually handicapped children in 
their classes. The questions served as an opening to more informal 
conversation to uncover the desired information. The interviews each 
lasted approximately thirty minutes. Three were held in the teachers' 
schools; the fourth was held in the teacher's home at her request. 

2. Instructional Specifications 
As the review of the literature pertaining to education of 
visually handicapped children progressed, instructional objectives for 
CARE 4 were written in terms of students' (classroom teachers') 



expected terminal behavior, and a proposed course content outline was 
composed along with items for the final examination. The need for a 
handbook became apparent as course preparation continued. The handbook 
was prepared during Product Devftlopment. A text for the course and a 
pamphlet were selected as off-line materials; these supplemented the 
handbook and on-line course material.^ 

Three outside experts were asked to examine the rationale, 
course objectives, content outline, final examination, and CARE 4 Hand- 
book and respond to the following questions based on Stake's suggestions 
for outside evaluation procedures: 

1. Is the rationale logical? 

2. Do the purpose and objectives meet the need presented in the 
rationale? 

3. Does the content relate to the purpose and objectives? 

4. Does the final exam cover the content? 

5. Does the final exam actually test the student's mastery of the 
course objectives? 

The outside experts were chosen because of their teaching and supervisory 
experience, their present professional positions, and their leadership 
roles in national professional organizations. One was the supervisor of 
programs for visually handicapped children in the Pennsylvania Depart- 
ment of Education, another a teacher of young deaf-blind children, and 
the third a supervisor of programs for visually handicapped for the 
Boston, Massachusetts City Schools. Materials were submitted to these 
evaluators once copies of the CARE 4 Handbook were available. 



In addition to the C ARE 4 Handbook . off-line materials included 
Teaching About Vision published bv the National Society for the Preven- 
tion of Blindness, inc.. New York, 1972; and "Helping the Partially 
Seeing Child In the Regular Classroom" available from the Pittsburgh 
Branch, Pennsylvania Association for the Blind, 308 South Craig Street, 
Pittsburgh, Pa. 



3. Item Tryout 

After sections of the rourse material had been written on 
author sheets provided by the CAI Laboratory but before they were sent 
to the Laboratory for programing, the course authors (the investigator 
and co-author) examined the CRT sheets, audio messages, image sketches, 
and slides for accuracy, consistency, wording, lay-out, flow within the 
total course, appropriateness to target population, and relevance to 
course objectives. Extensive revisions were made before these author 
sheets were sent to the Lab. 

After the CAI Laboratory staff had programed the course sec- 
tions , recorded the audio messages, and prepared the images, the inves- 
tigator went through course sections to check content and sequence of 
presentation in the dynamic on-line setting. 

Subject matter experts in the area of education of visually 
handicapped children and others in the general area of special education 
were asked to work through cours. sections and write their evaluation 
comments pertaining to course content, sequence, and quality on evalua- 
tion cards (Appendix G). Over several months, these evaluators spent 
varying periods of time on-line totaling from approximately one hour 
to over eight hours. 

CAI Laboratory staff members also recorded their comments on 
course presentation as they continued their technical revisions and made 
revisions specified by the authors. The Lab comments were turned over 
to the investigator for consideration and appropriate action. 

4. Product Development 
Evaluation comments nade early in the Instructional Specifica- 
tion step began to indicate the need for a handbook for students to use 



in conjunction with on-line course material. In addition, students had 
to have access to certain off-line information in order to complete 
the sixth chapter of CARE 4 on-line. The authors, therefore, prepared 
the CARE 4 Handbook referred to in the discussion of Instructional 
Specifications. 

To evaluate the CARE 4 Handbook and other off-line materials 
(above, p. 45, 1), copies of each were submitted to two post-master's 
students in special education. These students were to review the three 
publications giving consideration to the clarity of the Handbook chan- 
ters, the relationship of the Handbook content to the course objectives, 
the attitudes reflected toward regular classroom teachers and toward 
personr with visual impairments, consistency among the three publica- 
tioi^^ repetition ci content, and the relevancy of all three publica- 
tions tu the course objectives. In Appendix H are reported the specific 
questions these students were to answer and the comments returned by 
one of the students. 

The CAI Laboratory kept a record of the hours of on-line time 
spent by the authors and evaluators. Comments which had been collected 
from Lab personnel, subject matter experts, and special education per- 
sonnel were examined by the investigator and co-author. Revisions in 
the sequence, content, and format of CRT's and audio messages based on 
these comments were sent to the CAI Laboratory. Most content and tech- 
nical revi«;ions were completed before CARE 4 was sent to the Mobile Lab 
for the field test. 

5. Product Tryout 
CARE 4 was offered for one graduate credit through Penn State 
Continuing Education in August, 1972 while the Mobile Laboratory was 



located at The Woods School in Langhorne, Pennsylvania. Because of time 
constraints and because of the purpose for the tryout, the maximum num- 
ber of students allowed to register for CARE 4 was set at fifteen, 
specifically the first fifteen to register for the course. These fif- 
teen were to represent students in the target population for CARE 4 
since they would have themselves initiated the request to take the course 
and would be using the facilities in the Mobile Laboratory. Only after 
each student had completed CARE " was he permitted to start CARE 4. 

6. Product Revision 
After the completion of the field test, the investigator exam- 
ined student response records, on-line time records fcr each chapter and 
for the total course, print-outs of unanticipated responses to course 
questions, print-cuts of student comments made on-line by students who 
had branched themselves into the comment routine, and results of the 
final examination. On the basis of this information, further revisions 
were specified for CARE 4. Based on the comments received about the 
off-line materials, revisions were planned for the CARE 4 Handbook as 
well . 

7. Operations Analysis 
As the development work went on, the investigator identified 
steps that were essential for an author to take to make formative evalu- 
ation an integral part of CAI course preparation and to produce CAI 
course material in a form suitable for turning over to the CAI Lab for 
programing. These steps were reviewed with the CAI Lab staff members to 
uncover points where critical time delays occurred or where problems 



developed and points where specific activities facilitated course pre- 
paration. Forms and procedures thought to be useful to other course 
development projects were also identified. 



CHAPTER IV 



RESULTS OF FORflATIVE EVALUATION 

Formative evaluation for purposes of course improvement yields 
a mass of information in terms of both amount and content. The results 
of the formative evaluation of CARE 4 have been organized and v.'ill be 
reported here under the headings used in the Baker and Schutz cycle for 
Instructional product development according to the Product Development 
Rules stated in Appendix A of this document. 

A. Formulation 

The activities during the Formulation stage of CARE 4 included 
the preparation of a rationale for the development of CARE 4 (Appendix 
E), a review of the professional literature pertaining to the education 
of visually handicapped children (Appendix D), anH the completion of 
four semi -structured interviews with classroom teachers who had had at 
least one visually handicapped child in their classes. 

The replies to the interview questions (Appendix F) showed that 
these teachers were concerned about the availability of special mate- 
rials, the need for delivery of material :n time, the attitudes of 
other children toward visually handicapped rhildren, extra demands on 
teachers' time, and visual perception problems. Three teachers indi- 
cated a certain amount of frustration could be expected, particularly 
if materials were not available when needed. One mentioned that merely 
knowing that an itinerant teacher was available alleviated her worries. 
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Three stressed the need for patience. One warned of the danger of 
assuming that a child has had experience to give meaning to the words 
he uses. 

Based on a study of interview comments and drawing from prac- 
tical experiences as teachers of visually handicapped children, the 
investigator and co-author decided to: 

1. Stress early in the course the importance of teacher attitudes 
toward visually handicapped children and the possible effects 

of teacher attitudes on both normally seeing and visually handi- 
capped students. 

2. Emphasize to CARE 4 students that 

a. Special materials answer neither the needs of all visually 
handicapped students nor all the needs of any one visually 
handicapped student. 

b. Many special materials are available locally or through 
instructional rriaterials centers. 

3. Specify early in the course that CARE 4 is about education of 
partially seeing and blind children. 

4. Describe the role of the itinerant teacher. 

5. Explain the importance of concrete experiences to the language 
development of children with impaired vision. 

B, Instructional Specifications 
The investigator used the instructional objectives for CARE 4 
(Appendix C-2) as a guide in selecting and sequencing course material 
from the content outline which had been prepared after completion of 



the review of the literature. The content outline also served as a 
source of Items for the final examination. (See Appendix C-3 for an 
abbreviated form of the content outline.) 

Suggestions and Ideas from the teacher interviews discussed In 
this chapter under Formulation were Incorporated Into on-line materials 
In the following ways: 

1. A CRT message In the second chapter of CARE 4 addresses Itself 
to teacher attitudes that may be Inferred from patterns of 
behavior. Also In the second chapter after a section of educa- 
tionally relevant Information a CRT message points out that, 
just as parents* attitudes can influence normally seeing chil- 
dren, so can teachers' attitudes; and that the same is true even 
if the children have impaired vision. 

2. In the third chapter of CARE 4 students must work through a 
section in which they see that a child's needs may not be met 
just by special books or seating. The availability of special 
books and equipment takes up a major portion of the fourth chap- 
ter of CARE 4. 

3. One section of the first chapter of CARE 4 discusses the possi- 
ble implications of impaired sensory input channels in general 
and the visual input channel in particular. In the second chap- 
ter definitions for both partial vision and blindness are dis- 
cussed and various criteria used to identify each are presented. 
Emphasis is placed on the process of seeing and the components 
of visual functioning. 

The CAI Laboratory staff had suggested, as mentioned in Cnapter 
III of this document, that a handbook be prepared to provide students 



with a hard-copy of course -related information. In addition, the 
investigator's proposed design for the sixth chapter of CARE 4 called 
for certain data to be available off-line for students to use as they 
completed on-line material during a simulation exercise. For these 
reasons, plans were formulated for the CARE 4 Handbook , although no 
actual writing was done until the on-line course material had been com- 
posed, reviewed, and turned over to the CAI Laboratory for programing. 

The three outside evaluators who were asked to examine the 
course rationale, objectives, content outline, final examination, and 
Handbook returned their comments to the investigator while the Product 
Tryout activities were in progress. Their replies are reported in 
Appendix J. 

The three outside evaluators agreed that the rationale was sat- 
isfactory. £valuator 01 suggested that more compassion be expressed for 
the difficulty rural schools may have in finding and keeping teachers. 
Evaluator 02 thought details about length and amount of course credit 
should be added. Evaluator 03 felt the statement defining the intended 
target population might be broadened to take in city teachers. 

While all three evaluators judged the purpose and objectives to 
be adequate, they all mentioned the need for discussing resource per- 
sonnel outside the school to whom regular classroom teachers could make 
referrals and from whom they might request assistance. The evaluators 
seemed particularly concerned that resource persons for vocational coun- 
seling and basic mobility be identified. 

Each evaluator viewed the course content outline as meeting the 
purpose and objectives, but each offered items which might be appropri- 
ately inserted. These items included a distinction between acuity and 



diagnosis, the importance of peer attitudes, additional criteria for 
selection of materials, greater emphasis on utilization of low vision, 
additional causes of loss of vision, and suggestions for formulating a 
more useful definition of partial vision. 

The final examination was described as sufficiently long and 
comprehensive to cover the course materials. Two evaluators commented 
that they liked the questions which set up hypothetical situations call- 
ing for students to make decisions. Two concluded that the best exami- 
nation would be the observation and report of any changes in teacher 
behavior in the classroom related to having completed CARE 4. 

C. Item Tryout 

The activities completed under Item Tryout included author revi- 
sion, both before material was programed and after it was on-line, and 
on-line evaluation by subject matter experts and special education per- 
sonnel. The author revision prior to programing preceded all other 
activities in Item Tryout. 

The evaluators were scheduled o^er a five month period. Revi- 
sions viewed as necessary after examination of each evaluator's comments 
upon the completion of his on-line time were made as soon as possible. 
Those evaluators who were scheduled later in the evaluation period, 
therefore, actually were evaluating an already-revised version of 
CARE 4. 

A report of the author revisions and evaluators* comments fol- 
lows. The Item Tryout section concludes with illustrations of the 
evaluation and revision process. 



1 . First Author Revision 

After the completion of original authoring of a course section 
varying in length from six format sheets with accompanying audio mes- 
sages and image sketches to an entire chapter, the investigator and co- 
author together examined the section for accuracy, consistency, wording, 
lay-out, flow within total course, appropriateness to target population, 
and relevancy to course objectives. 

For example, the investigator and co-author spent approximately 
three hours revising the fifth chapter of CARE 4 before sending it to 
the CAI Lab. They made over twenty changes in the original version by: 

1. Adding seven examples to clarify statements and a question to 
assess student understanding; 

2. Deleting three CRT messages and parts of two audio messages; 

3. Rewriting six CRT and audio messages to state ideas more 
clearly; 

4. Substituting more appropriate words or phrases for less precise 
ones in five CRT and audio messages; and 

5. Anticipating additional answers to a CRT question and writing 
suitable responses for those answers, 

2. Second Author Revision 
The review of CARE 4 on-line began as soon as course material 
had been debugged at the CAI Lab. The investigator and co-author spent 
a total of 78.5 hours on-line on six separate occasions before and dur- 
ing the evaludtors* on-line work. 
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Table 1 presents a summary of the kinds of changes that were 
made according to the medium to which each pertained (CRT, audio mes- 
sage, or image), the type (technical point, error, content remark, or 
compliment), and the chapter to which each applied. 

3. Subject-Matter Expert Evaluation 
Six subject-matter experts acted as CARE 4 students and recorded 
their comments on comment cards as they progressed through the course. 
Table 2 shows the summary of their comments according to the medium to 
which each referred, the type, and the chapter to which each pertained. 

4. Special Education Personnel Evaluation 
Six individuals with background in special education but with 
emphasis in areas other than the education of visually handicapped chil- 
dren reviewed course sections and recorded their comments on the comment 
cards. Table 3 summarizes the comments made by this group of evaluators 
according to the medium to which each applied, the type, and the chapter 
to which each referred. 

5. Illustration of the Revision Process 
Comments collected from the subject-matter experts and special 
education personnel were separated into remarks about technical pro- 
graming aspects of the course, comments regarding course content, iden- 
tification of errors, or compliments pertaining to either course content 
or manner of presentation. Technical problems were usually turned over 
immediately to Lab personnel; some, however, required author decisions 
as to what, if any, changes were indicated in presentation, sequence, or 
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mode. All other comments were reviewed by the Investigator who speci- 
fied what alterations were to be made in light of those comments. For 
example: 

Technical Comment . --Near the beginning of the first chapter of CARE 4, 
students hear a person pull a chair out from a table, sit down, move the 
ch:ir closer to the table, insert paper into a typewriter, and type 
several lines. The students are to identify what they have heard. The 
ancitipated correct reply is some form of the word 'type' and any one of 
the set: adjust, move, pull, push, chair, seat; insert, roll paper, 
typing paper; bell, ring bell. A student who types in only one antici- 
pated correct response is asked to listen to the audio message again to 
see if he can identify any other sounds. If a^ter the second time the 
student has not responded correctly in terms that the system can recog- 
nize, he is told the 'correct' answer and is reminded how important 
auditory cues are to children with limited or no vision. 

Evaluator #5 did not respond correctly after the second time. 
Because of a technical error, she heard the message five times before 
she finally called for assistance. Her frustration and annoyance were 
apparent, and she expressed a feeling of personal failure at CAI. The 
error was 'immediately explained to her, a note was made for the pro- 
gramer to check the directions for progression, and several more answers 
were made acceptable to the system on the basis of the replies Evalu- 
ator #5 had been giving. Evaluator #5 was encouraged to continue and 
later reported that she was extremely impressed with CAI as a means of 
learning and that she had recovered from her initial bad experience. 
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Identification of Error . --In the section describing the sources of 
special materials for students in Pennsylvania, an image displays a map 
of Pennsylvania on which the locations of three instructional materials 
centers are marked. Addresses accompany each location. Evaluator E, 
who happened to be an assistant director of one of the centers, indi- 
cated that the suite number in one address was incorrect; the error 
would not probably cause any loss of mail, but it might cause a delay. 
The investigator made a rote of the error for correction when the CARE 4 
image reels are revised. 

Course Content .— In the second chapter of CARE 4, students are shown a 
teacher observation report of a child. From the report they are to 
select the behaviors described which might indicate the possibility of 
the child's having limited vision. Evaluator A pointed out that the 
report actually presented only one description of behavior; the other 
information was the child's self-report of his visual behavior. The 
child's self-report information was deleted from the teacher observation 
report, and descriptions of the child's actual behavior were inserted. 

Because CARE 4 was originally planned for teachers in 
Pennsylvania, one section in the original version of the course 
described sources of materials and equipment in Pennsylvania. It was 
recently reported that the Mobile Laboratory has been scheduled to 
travel to sites outside Pennsylvania. Branches have, therefore, been 
written to take CARE 4 students who are not Pennsylvania residents 

I 

through similar information but of a more general nature. | 

i 

Compliment . —Several complimentary remarks were made about both sections 
of content and manner of presentation. 
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In the first chapter of CARE 4 students see parts of an ele- 
phant appear on the CRT while an audio message explains how a person 
with limited visual input might build an idea of 'elephant.' At least 
five evaluators wrote that they thought the animation on the CRT rein- 
forced the discussion on the auditory message. 

In the second chapter of CARE 4 the introductory statement on an 
audio message stresses that using t' '-^ eyes cannot hurt them or cause*^^ 
further impairment except in rare instances. Evaluators #2 and #3 com- 
mented that the discussion made an excellent point in favor of sight 
utilization. 

D- Product Development 

Table 4 summarizes by chapter the components of CARE 4 after all 
revisions were completed prior to the field test. The total number of 
student/computer interactions represents the total number within the 
course. When one considers the variety and number of programing strate- 
gies used in individualized instruction in CARE 4, it is unlikely that 
any one student would be confronted with every possible interaction ' 
written into the course. 

Table 5 shows the final sequence of CARE 4 as it was delivered 
to the field test students. The segment numbers and labels indicate the ' 

relative location of chapters on the storage discs assembled for the 3 

f 

computer. ^ 

Once course materials were prepared for the CAI Laboratory, work I 

I 

began on the CARE 4 Handbook . The version of the Handbook which was i 

f 

submitted to the outside evaluators and used for the Product Tryout con- 
tained seven chapters, one corresponding to each of the seven chapters 



TABLE 4 
COMPONENTS OF CARE 4 







Chapter 


CRT s 


Aud 1 0S 


Images 


Student/ Computer 
Interactions 


Coursewriter II 
Statements 


I 


49 


.9 


12 


12 


3806 


II 


216 


62 


56 


137 


9883 


III 


54 


16 


7 


33 


2746 


IV 


60 


15 


35 


31 


2797 


V 


67 


16 


9 


41 


3046 


VI 


50 


7 


10 


31 


2287 


VII 


35 


8 


5 


11 


1595 


VIII 


66 




6 


67 


4048 


Total 


597 


143 


140 


363 


30208 
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TABLE 5 
CARE 4 COURSE SEQUENCE 



Chapter 


Title 


Segment 


Labels 




How to 


0 


aa01/ 
bib01a 


1 
1 


in truQUu 1 1 un 




ba01a 


2 


Identification of Educationally 
Relevant Characteristics of Visu- 
ally Handicapped Children 


2 


ca01a 


3 


Construction of Instructional 
Objectives 


3 


da01a 


4 


Selection of Instructional Materials 


4 


ea01a 


5 


Arrangement of Classroom Environ- 
mental Conditions 


5 


fa01 


6 


Design of Instructional Procedures 


6 


ga01 


7 


Utilization of Appropriate Techniques 
for Evaluating Performance 


7 


ha01 


8 


Summary and Final Exam 


8 


ja01 




Drop; Sign-off 


126 


drop/ 
fini 



on-line, but excluding the final examination. Excerpts from CRT's and 
audio messages and Illustrations taken from the Images made up the con- 
tents of the Handbook . 

The evaluation of the CARE 4 Handbook and other off-line mate- 
rials was completed by one post-master's special education student (see 
Appendix H for his report)c The Handbook had also been given to the 
outside evaluators to examine (see Appendix J-3). Based on the comments 
of these evaluators the off-line materials were judged to be suitable 
for use In the field test, but suggestions for revisions were noted for 
future reference. 

E. Product Tryout 

During August, 1972 fourteen students registered to take CARE 4 
for credit while the Mobile Lab was located at the Woods School. One 
Lab proctor also signed up for xhe course but not for credit. Of these 
fifteen students, thirteen completed the course and two dropped It 
before spending any time on-line. 

Table 6 shows the final exam scores for the thirteen students 
who finished CARE 4. Out of a possible 107 points, the mean was 92.5 
and the standard deviation 6.2. Seven scores were above the mean and 
six fell below. 

An examination of the student response record revealed how many 
students responded with each of the anticipated correct and Incorrect 
answers and how many gave unanticipated answers. The CAI Laboratory 
provided the Investigator with a computer printout of the unanticipated 
replies given to specific questions. These replies will be used In 
later revisions of CARE 4 to broaden the restricted vocabulary of 



TABLE 6 
CARE 4 FIELD TEST RESULTS 



Student Final Exam 

Code Score 



Proctor 


103 


VWBC 


93 


VWGD 


100 


VWJH 


86 


VWLM 


100 


VWSC 


91 


VWAT 


98 


VWWK 


94 


VYWJ 


86 


VYZR 


82 


VZWC 


94 


VZFR 


90 


VZSS 


86 



X = 92.5 
SD = 6.2 
Highest Possible Score = 107 



correct anticipated keyboard answers. These replies will also show 
unanticipated wrong answers which may require recognition with a spe- 
cial remedial response. 

Within the first, second, and fourth chapters of CARE 4 are 
thirteen options which students may select In order to review material 
previously seen or to receive more Information about a particular topic 
Student options are recorded In Appendix K. Table 7, which surmarizes 
the number of times each option was chosen according to information 
reported on the student response record, reveals that twelve of the 
thirteen options were selected by at least one student, and three 
options were chosen by at least six students. 

In addition to a response record for the fifteen students, the 
computer stored comments made by students who chose to branch into the 
comment routine. A total of three students offered sixteen comments, 
mainly about audio quality, acceptable recognized responses, and space 
allowed for typing in answers. These comments are reported in 
Appendix L. 

F. Product Revision 
The investigator gathered information from the following rec- 
ords to plan the revision of CARE 4 at the completion of the field test 

1. Student response records; 

2. Printout of unanticipated responses; 

3. Student on-line comments; 

4. Final examination results 

5. Comments by evaluators of the off-line materials; and 



TABLE 7 
STUDENT OPTIONS TO BRANCH 



Number of Students 

Chapter Location Selecting Option 



CARE 4-1^ 



ba25a 


6 


CARE 4-2 




ca10a 


6 


cd03a 


1 


ce38a 


2 


ce49a 


6 


ce64b 


1 


ce70b 


1 


ce97b 


2 


cf05a 


2 


CARE 4-4 




ec02a 


4 


ec04a 


7 


ec26a 


0 


ec33b 


5 



CARE 4-1 means Chapter I of CARE 4. The alpha- 
numeric code, ba25a, indicates a specific CRT or frame 
within Chapter I where the branching option was presented. 



6. Comments by the outside evaluators of the rationale, objectives, 
content outline, and final examination. 
Any decisions to revise course material were based upon a number of con- 
siderations, not just specific objective criteria. Such considerations 
included course objective, professional backgrounds of evaluators, 
technical complexity of suggestions, similarity of other comments, fre- 
quency, and the relative importance of the specific comment in light of 
other comments. 

Based on a study of the comments offered by the evaluators and 
the records of student progressions through CARE 4, these changes were 
indicated: 

1^ Re-recording of audio messages .--All the evaluators and two 
field test students commented on the quality of portions of 
audio messages. In addition to changes in the content and 
length of some messages, and a request^ for another voice to do 
the recording, several new messages were inserted where explana- 
tions were found to be incomplete. 

2. Division of Chapter Two . —A glance at Table 4 shows that the 
second chapter of CARE 4 is longer than any other chapter in the 
course. Several evaluators commented on the length and sug- 
gested that it be divided. The content allows division into 
four chapters with the insertion of very little additional mate- 
rial for the sake of continuity. The proposed separations would 
create the following chapters: Identification of Visually 
Handicapped Children, Collection of Educationally Relevant 
Information About Visually Handicapped Children, The Process of 
Seeing, and Common Causes of Limited Vision in School Children. 
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3. Use of comment routine more frequently . '•'•Although all the field 
test students had completed CARE 1 and knew how to use the com- 
ment routine, few actually chose to comment. This may have been 
because only once in CARE 4 were they invited to make comments. 
It may have also been because these students had no comments to 
offer. In any event, the investigator decided to have inserted 
reminders to students that the comment routine was available 
and that course authors would carefully consider any comments 
students made. 

4- Additions to anticipated responses . —The student response record 
indicated to the investigator that field test students had 
responded to several questions in the course with acceptable 
answers which were subsequently treated as unrecognized, and 
possibly incorrect, because the words used to communicate the 
answer had not been anticipated and stored in the program. 

For example, in the second chapter of CARE 4 after a 
rather detailed discussion of clinical and functional defini- 
tions used to differentiate partial vision and blindness, stu- 
dents are asked five questions as a review. The first is: 

1. Definitions describing visually handicapped children are 
usually one of two kinds. Name the kinds. 

The anticipated answers considered correct are "clinical" and 

"functional." No anticipated incorrect answers are specified 

for this question; all answers not recognized are counted as 

wrong. Five students, however, gave "partially seeing" and 

"blind" as answers. Although counted wrong, they are reasonable 

wrong answers considering the phrasing of the question and the 



course material that had preceded the question. Among the 
alternatives available, the investigator chose to let the ques- 
tion remain unchanged, since seven of the twelve whose answers 
were recorded replied correctly; add "partially seeing" and 
"blind" as anticipated incorrect answers; and write an appro- 
priate response to correct any student who gives those answers. 

Printouts of unanticipated responses show where typing 
errors cause replies to be counted as incorrect. The antici- 
pated correct answer to another question in the second chapter 
is "right" which has been reduced to the key letters "ri" and 
"rt." One student typed "fight" which was tallied as wrong. 
There is no provision presently, however, to ascertain whether 
the student knew he was wrong for the right reason. Presumably, 
since he was asked to answer the question again, he would notice 
the typing error. 

Content changes .— A review of the evaluation comments uncovered 
a number of suggested additions to course content but no dele- 
tions. Because CARE 4 is to remain a one credit course, the 
total size of all additions to course material must be limited. 
Changes currently under consideration are: 

a. Addition of options to request more information cbout 
vision screening, sight utilization, and the use of the 
abacus. 

b. Revision of section on clinical and functional descrip- 
tions of limited vision. 

c. Substitution of frequently missed final exam questions 
with more hypothetical situations requiring students to 
make decisions. 



d. Revision of Handbook , particularly if the second chapter 
CARE 4 is divided into four chapters; in addition, other 
chapters need editing. 

G. Operations Analysis 

As course development and formative evaluation steps proceeded, 
it became apparent that the completion of certain author activities 
hinged upon the completion of other activities. The investigator iden- 
tified the activities which seemed to be critical and arranged them in 
a flowchart as illustrated in FIGURE 5. Only those events which must be 
completed by the author before materials are submitted to the CAI Labo- 
ratory are indicated. 

The flowchart of author events was given to the educational pro- 
gramer who managed the technical aspects of CARE 4 development. She 
pointed to events five, six, and eleven as crucial to the completion of 
any CAI project: an author must know how the CAI system for which he is 
writing operates, he must know axactly what his objectives are for his 
course, and then he must start writing course material. Authoring mate- 
rial, although a demanding and time-consuming task, requires only a 
small part of the total course development time. (Appendix M summarizes 
the steps which must come after course material is submitted to the CAI 
Laboratory at The Pennsylvania State University.) Programing and 
debugging consume many hours of CAI Laboratory time in addition to time 
for the preparation of audio tapes and image reels. Table 8 sunmarizes 
the on-line times of the course authors, technical programers, subject 
matter experts, special education personnal, and others. The times 
reported in Table 8 do not reflect the use of the 1500 System in the 
Mobile Laboratory. 



Possible need for a CAI course identified. 

2. Preparation of CAI course rationale completed. 

3. Review of professional and research literature related to course 
completed. 

4. Representatives of target population interviewed. 

5. Practice at CAI student terminal to learn system capabilities 
completed. 

6. Course objectives prepared. 

7. Final exam prepared. 

8. Course outline prepared. 

9. Comparison of course objectives, final exam, outline, and 
rationale for consistu.icy. 

10. Outline, objectives, final exam and rationale submitted to outside 
evaluators. 

11. CRT's written. 

12. Audio messages written. 

13. Image sketches completed. 

14. Photographs taken. 

15. Evaluation of nutside evaluation comments completed. 

16. Modifications in course rational, objectives, final exam, and out- 
line completed. 

17. Course material completed. 

18. Author revisions on paper completed. 

19. Clean copy for CAI Lab completed. 

20. Clean copy mailed to CAI Lab. 




Fig. 5. Autho events prior to submitting course material to 
CAI Laboratory. 
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The educational programer emphasized the Importance of an 
author's beginning to write course material as soon as possible since 
an CAI Laboratory work is tied to whatever the author specifies on the 
authoring sheets. She stressed the necessity of adhering to realistic 
deadlines for the completion of author activities 

In analyzing the completion of CARE 4 activities, the educa- 
tional programer and investigator agreed that course material should 
have been submitted to the CAI Laboratory sooner than It was In order to 
prevent the rush of activity that occurred as the course preparation 
time came to an end. Since time, which could have been devoted to on- 
line revision of course material had material been submitted earlier, 
had to be dedicated to programing, the quality of the final product may 
have suffered. 

H. Procedures and Forms Found Effective and 
Suitable for Possible Inclusion In a 
Manual for a CAI Course Author 
to Use While Conducting A" 
"Formative Evaluation 

As a result of having completed the formative evaluation activ- 
ities and of having reviewed the operations analysis of CARE 4, the 
Investigator identified certain procedures and forms which were 
effective In the preparation of CARE 4. These procedures and forms may 
be useful to other CAI course authors preparing course material for th 
1500 System while at the same time conducting formative evaluation 



Times differ from those reported undei' Item Tryout because 
Individuals besides those described In this document also spent time on 
CARE 4 for other purposes, one of which was to learn how the 1500 Sys- 
tem operates, and their times were Included In the total tlrnGS^ 



activities. Some discussion of these procedures and forms might be 
appropriate to include in a manual for authors of CAI course material. 

Use of the Baker and Schutz cycle . -The Baker and Schut.z cycle for 
instructional product development provided a framework within which the 
investigator could schedule procedures appropriate for CARE 4 course 
development. This model for formative evaluation was judged to be quite 
satisfactory for purposes of formative evaluation of CARE 4 course 
rnaterial . 

On-line time to learn the 1500 System capabilities .-Time spent on-line 
learning the capabilities of the 1500 System and how others had 
exploited them provided the investigator with ideas about how system 
capabilities might be used effectively and illustrated the advantages 
and disadvantages of certain formats for content presentation and ques- 
tioning. The investigator also had the opportunity to see various forms 
cf feedback to both correct and incorrect student responses. 

Author revis ion prior to submission of material to CAI Laboratory -The 
author revisions prior to submitting course material to the CAI Labora- 
tory eliminated many corrections and changes which would have consumed 
on-line time to make or which would have necessitated the programer's 
discussing with the author just what was the intent of certain author 
instructions. 

Use of evaluation comment cards — Evaluators reported that use of com- 
ment cards with space to identify the label and medium made the evalua- 
tor's task of recording comments less time-consuming. The investigator 
found that the cards reduced the investigator's time required to iden- 
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t1fy the material to which each comment referred. Laboratory personnel 
adopted the same comment card format the Investigator had set up for 
course evaluators, another apparent indication of the usefulness of the 
card. Prior to that time blank cards had been provided i^cr evaluators 
by the Laboratory. Comment Card A, described by some evaluators as too 
restrictive, was chosen only occasionally by three of the twelve on-line 
evaluators. All other comments were written on Card B (Appendix G-2 and 
6-3). 



CHAPTER V 



SUMMARY, CONCLUSIONS, LIMITATIONS, AND SUGGESTIONS 
FOR FURTHER INVESTIGATION 

This examination and application of formative evaluation by 
an author during the preparation of CAI course material required the 
completion of a variety of activities whicu will be summarized in 
this chapter. The conclusions drawn from the results of the forma- 
tive evaluation activities will also be reported along with certain 
limitations of the study and several suggestions for further investi- 
gation. Attention is called to the fact that, because of the 
descriptive nature of this study, the conclusions reported here and 
the suggestions for further investigation based on those conclusions 
reflect the experiences of the investigator during the particular 
activities as described in this document, 

A. Summary 

The purpose of this study was to investigate and report an 
author's use of formative evaluation during the preparation of a 
course designed for presentation to students via computer-assisted 
instruction. In order to complete the study the investigator exam- 
ined models for formative evaluation of course material and then, 
after conferring with dissertation committee members, selected a 
model appropriate for an author tc use during the preparaM'on o? CAI 
course material. Using the model selected, she conducted the 
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formative evaluation of a CAI course entitled "Education of Visually 
Handicapped Children" and identified information suitable for pos- 
sible inclusion in a manual for other CAI course authors to follow 
when conducting formative evaluation during the preparation of 
instructional programs. The model chosen, the Baker and Schutz 
cycle for instructional product development, called for seven general 
procedures: formulation of a rationale for the proposed course; 
specification of the instructional objectives, course content, and 
final examination; tryout of small sections of the course; develop- 
ment and assembly of the entire instructional product; product tryout 
with a small group of 'Students representative of the target popula- 
tion; product revision subsequent to analysis of product tryout 
results; and operations analyris of the complete development program 
to identify strengths and wcdKnesses in the formative evaluation pro- 
cedures. From the Stake model for formative evaluation tne investi- 
gator added to the general procedures the outside evaluation of 
rationale and course components. 

The completion of the :Drocedures necessitated interviews with 
regular classroom teachers, evaluation of rationale and course com- 
ponents by outside evaluators, evaluation of on-line course material 
by subject matter experts and special education personnel, and two 
author revisions of course material. In addition, a field test wit'i 
fifteen students was conducted in a mobile CAI Laboratory operated by 
The Pennsylvania State University. At the conclusion of the study, 
revisions were cp*>cified for both on- and off-line course material, 
and procedures and forms were identified that possibly could be 



included in a manual for other CAI course authors to follow while 
conducting formative evaluation activities during preparation of CAI 
course material . 

B. Conclusions 

Based on the completion of the examination and application of 
formative evaluation by an author during the initial preparation of a 
CAI course, the investigator drew the following conclusions: 

1. The Baker and Schutz general model for instructional product 
development as adapted for this study is appropriate as a 
guide in designing formative evaluation procedures for CAI 
course preparation. 

2. Formative evaluation procedures can be carried out by a CAI 
course author. 

3. Weaknesses in course material can be easily identified when 
evaluators specify precisely where difficulties arise and 
when student response records reveal exactly what questions 
students miss and what incorrect responses they offer. 

4. Results of formative evaluation procedures can assist the CAI 
course author to identify errors and weaknesses in CAI course 
ma^-erial before any students take the course for credit. 

5. Initial success with CAI student terminal equipment is impor- 
tant in order to maintain a student's confidence and desire 
to continue with the course; system delays, technical errors, 
and unnecessary program loops detract from the efficiency of 
the system and seem to affect student attitudes toward CAI as 
a means of learning. 
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C, Limitations of the Study 
The investigator in reviewing the forma^-ive evaluation pro- 
cedures as they actually were performed noted several limitations in 
the procedures of this study. 

1. The investigator did not make formative evaluation an 
integral part of course design and development in a system- 
atic manner until several months into the CARE 4 project. 
This resulted in inefficient use of the CAI Laboratory staff 
time in the early stages of course development and cut into 
author revision time near the end of course development. 

2. Reports from outside evaluators v/ere received too late to be 
used in revising CARE 4 for presentation to students during 
the field test in Langhorne. They were, however, available 
for use during Product Revision. 

3. No accurate time records were collected from field test stu- 
dents in the Mobile Laboratory. As a result, no reliable 
estimate of the average amount of time students took to com- 
plete CARE 4 could be made. 

Copies of Teaching About Vision were distributed through a 
bookstore located at some distance from the Mobile Laboratory 
site in Langhorne; and it was reported that few CARE 4 field 
test students purchased this book, even though the book was 
listed as a required text for the course. The student 
response record for the second chapter of the course may 
reflect this fact since it was assumed by the authors that 
students would refer to this text for review of information 
necessary to complete that chapter on-line. 



4. 
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5. Only four classroom teachers were interviewed during the formu- 
lation activity because of the constraints imposed upon the 
study by time, budget, and the U. S. Office of Education regu- 
lations on conducting interviews without prior approval of the 
interview schedule by the U. S. Office of Education. 

6. This study had to be completed during the time specified and 
the budget agreed upon by the cooperating universities before 
the exact procedures for the study had been determined. As a 
result, some of the procedures such as teacher interviews had 
to be arranged in the most expedient way rather than in the 
most helpful way for purposes of the study, 

D. Suggestions for Further Investigation 
Suggestions for further investigation are grouped under three 
major headings: CARE 4. CAI course preparation, and formative evalu- 
ation during course preparation for the IBM 1500 System. 

1. Suggestions for Further Investigation 
Relating to CARE 4 

The following suggestions for additional consideration of 
CARE 4 include: 

a. Preparation of a pretest . -ComDari«;nn of pretest and posttest 
scores of students could indicate whether in fact the students were 
increasing their mastery of course material as a result of com- 
pleting CARE 4 or whether they knew much of the course material 
before starting the course. 



b. Comparison of fi.ial examination scores of students receiving 
different amounts of on- and off-line course material .—Posttest 
scores of three groups of students might be compared: stuHents who 
had completed CARE 4 on-line and who also had used all off-line mate- 
rials, students who had completed CARE 4 on-line but who had used no 
off-line materials other than those necessary to complete on-line 
material in the sixth chapter of the course, and students who had 
read only the off-line material. Such a comparison might show 
differences in the relative importance of off-line materials, on-line 
materials, and their combination in preparing students to meet the 
course objectives. 

c. Consequential evaluation of CARE 4 .— Such an evaluation could 
determine whether completion of CARE 4 had a / effect on the behav- 
iors of teachers when they were in their classrooms working with 
children. 

d. Effects of having completed CARE 4 on working relationships 
between itinerant teachers and classroom teachers .— The working 
relationships between itinerant teachers having completed CARE 4 and 
regular classroom teachers having completed CARE 4 may be influenced 
by the itinerant teachers' knowing the kinds of information the 
classroom teachers had acquired. Such a study as this would require 
itinerant teachers to take CARE 4 even though the target population 
for the course was regular classroom teache»-s. 
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2. Suggestions for Further Investigation of 
Procedures for CAI Course Preparation 
for the IBM 1500 Instructional System 

A study of student response records as well as a review of 

the decisions which the investigator and co-author Lad to make 

regarding the effective manner in which to present course material to 

students prompted many questions about how to design and sequence 

course material. Out of these questions came these suqoestions for 

further investigation. 

a. Investigation of the differential effects of audio, audio-visual, 
and visual sequences on student learning of small course sections .— 
Students may prefer one sensory input channel over another or a com- 
bination of two; further study may or may not sho'^ that their prefer- 
ence is their most efficient channel for learning. Results might 
also suggest that one channel or combination is consistently more 
efficient for learning than another or that the level of complexity 
of course material must be considered when deciding how to present 
course material . 

b- Development of inductive and deductive instructional sequences .— 
Further study may indicate that some students learn more efficiently 
when course material is presented either inductively or deductively. 
Other factors such as intellectual ability, age, and complexity of 
course material may also be identified as influencing learning 
efficiency. 

c. Effects of providing student options either to branch into addi- 
tional or to review previously seen information . --Although tutorial CAI 
courses can be designed to provide some student control of movement 
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through the course, most decisions concerning progress through the 
course are made by the author and are programed into the computer 
directions. Further study may reveal among other things that students 
prefer to be given as much control of their progress through the 
course as Is possible or that the desire for control is related to 
other factors. 

d. Effects of different levels of critical thinking skills on 
achievement in CAI courses .— It may be that success in CAI courses Is 
related to skills Involved in critical thinking as measured by 
instruments such as the Watson-Glaser Critical Thinking Appraisal. 
Students may demonstrate a range of abilities to analyze and 
synthesize information and to make decisions based on that informa- 
tion. There may be a relationship between these abilities and 
achievement on specific CAI courses. 

e. Usefulness of Flowchart of Author Events to other CAI course 
development projects .— The author events identified as Important to 
the completion of CARE 4 may or may not be Important in other CAI 
course development projects. 

f . Usefulness of the Baker and Schutz cycle for the development of 
instructional products to designers of other CAI course material . 
The Baker and Schutz model is presented as a general model. Further 
application of the model by CAI course authors could Increase the 
amount of data regarding the usefulness of the model in CAI course 
design. 



9- Usefulness of other models for formative evaluation to designers of 
other CAI course material . — It was the purpose of this study to -select 
only one model for formative evaluation to use during the development of 
CARE 4. Because the Baker and Schutz model as modified was judged appro- 
priate at the conclusion of the study does not preclude che possibility 
that other models might also be appropriate. Other models such as those 
reviewed in Chapter II of this document should be tried to help deter- 
mine their usefulness in CAI course design. 

h. Investigation of factors in the affective domain which influence 
student performance on CAI co urses. --Factors such as motivation, atten- 
tion span, connotations included in course material, physical condition 
of studei t, and mode of presentation may have some interrelationship 
with the performance of students on CAI courses as well as on attitudes 
toward CAI as a tool for learning. 

3. Suggestions "or Further Investigation 
of Formative Evaluation 

Areas for further study of formative evaluation include: 
3- Usefulness of the Baker and Schutz cycle for instructional prod- 
uct development .— Application of this model to the design of 
instructional programs other than CAI courses could reveal how effec- 
tively the model can be adapted for a variety of product development 
projects. 

b. Effects of having completed formative evaluation activities on 
the instructional product development skills of those perr-ons 
involved in formati " " evaluation .— A study of the instructional pro- 
duct development procedures and skills of those having performed 



formative evaluation activities could indicate if any effects of 
having completed those activities influence procedures and skills 
used in other instructional development projects. 
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APPENDICES 



APPENDIX A 



A REVIEW OF PRODUCT DEVELOPMENT RULES 



FORflULATION 

F:l. The extensi veness of a proposed product's justification should 
be commensurate with the importance of the product. 

F:2. Excessive time should not be spent in formulation. 

F:3. In justifying the development of the new product, make certain 
there are no competing products of high quality. 



INSTRUCTIONAL SPECIFICATIONS 

IS:1. All instructional objectives should be stated in terms of the 
learner's post-instructional behavior. 

IS: 2. En-route and entry behaviors should also be described behav- 
ioral ly in the instructional specifications. 

IS:3. Criteria for judging the adequacy of the learner's response 
should be specified. 

IS;4. A clearly specified method for detemiing learner affect 

toward the completed instructional product should be specified. 



ITEM TRYOUT 

IT:1. The criterion test must be completely prepared prior to the 
development of the instructional product. 

IT: 2. Measures of the entry and en-route behaviors should be con- 
structed during the item tryout stage. 

IT:3. Prototype items should not deviate from the behaviors described 
in the instructional specifications. 



Robert L, Baker and Richard E Schutz, eds., Instructional 
Product Development (New York: Van Nostrand Relnhold Company, 1971), 
pp. 167-68. 
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IT:4. Prototype items should be tried out with a small number of 
learners first, later with a larger number of learners. 



PRODUCT DEVELOPMENT 

PD:1. Supply the learner with appropriate practice during an instruc- 
tional sequence. 

PD:2. The product should provide the learner with the opportunity to 
obtain knowledge of results. 

PD:3. The instructional product should contain provisions for pro- 
moting the learner's interest in the product. 

PD:4. Avoid the development of an inflexible strategy in approaching 
product development tasks. 

PD:5. If teachers are involved in the instructional process, make 
their participation as replicable as possible. 

PD:6. In general, adopt a "lean" programming strategy. 

PD:7. If the product is to be used in the classroom, develop it so 
that teacher attitudes toward the product will be positive. 

PD:8. Selection of the instructional medium should be made in light of 
the desired instructional objectives, intended target popula- 
tion, cost, and other relevant considerations. 

PD:9. The time devoted to the development of the product should be 
commensurate with the importance of the product. 



PRODUCT TRYOUT 

PT:1. Avoid an extremely small or extremely large number of learners 
when field testing the product. 

PT:2. Verify that the procedures associated with the use of the pro- 
duct result in a replicable treatment. 

PT:3. Data from field trials should be efficiently summarized for use 
by those who will revise the product. 

PT:4. Those involved in field testing the product should collect 

data; they should not, themselves, engage in drawing inferences 
from the data. 
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PRODUCT REVISION 

PR:1. Base product revisions on legitimate inferences from fielJ test 
data. 

PR:2. The primary inferences regarding product revision shoulJ be 
made from criterion data 

PR:3. Learner response data during the program should be co/isidered a 
valuable source of cues for product improvement 

PR:4. No loss of face for the initial developer should be associated 
with revisions of an instructional product. 



OPERATIONS ANALYSIS 

0A:1. Operations analysis should be performed at the conclusion of all 
systematic development of instructional products. 

0A:2. The operations analysis should be written and transmitted to 
some central repository 



APPENDIX B 



IBM 1500 INSTRUCTIONAL SYSTEM 

Each student station in the IBM 1500 Instructional System in 
both the CAI Laboratory on the campus of The Pennsylvania State Univer- 
sity and in the Mobile Laboratory is equipped with a cathode ray tube 
or CRT, light pen, typewriter keyboard, audio playback unit, and image 
projector. All of these components operate under program control. 

Information can be presented to the student on the CRT, on pre- 
recorded audio messages, and/or on images shown on the image projector. 
The CRT screen has an area equivalent to 640 display positions or six- 
teen horizontal rows and forty vertical columns. Audio messages can be 
programed to pause while displays appear on the CRT or while images 
appear on the image projector. The image projector can hold a one-thou- 
sand image lemm microfilm reel. 

Students can respond to the system either by typing responses on 
the keyboard which appear immediately on the CRT or by pointing to spe- 
cific areas on the CRT with the light pen. Responses entered either on 
the keyboard or with the light pen can be processed and answered appro- 
priately by the computer according to author instructions. 

The Central Processing Unit or CPU with its support system per- 
mits the dynamic interaction between the student and the instructional 
system. The CPU in the 1500 System can accommodate up to thirty-two 
student stations. It contains 32,786 sixteen bit words of core storage. 
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APPENDIX C-T 
CARE 4 

CARE 4 was developed at the University of Pittsburgh in con- 
sortium with The Pennsylvania State University as a one-credU course 
for inservice training of regular classroom teachers in rural areas who 
have or may have visually handicapped children enrolled in their 
classes. The initial writing and revising of CARE 4 was done on author 
sheets in a format specified by the CAI Laboratory to insura dear com- 
munication between author and programer. (See Appendix H for sample 
author sheets.) 

Although most of the images for CARE 4 were drawn and prepared 
by the CAI Laboratory graphic artist according to author specifications, 
some images were processed from 35mrn color slides which show actual 
children with limited vision in the school environment. Other images 
were processed fv^om slides showing materials and equipment frequently 
used by many school children who have limited or no vision. 

Audio messages in the course vary in length from several seconds 
to several minutes. Most present information to students or elaborate 
upon information displayed on the CRT or on an image. Other messages 
indicate whether students* responses are correct or incorrect and why. 
On still others, students hear conversations which simulate discussions 
between two teachers and between a teacher and parent. 
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The course required 30,208 Coursewriter II statements to pro- 
gram for the IBM 1500 Instructional System. The course preparation 
time lasted fifteen months. Student time to complete the course is 
approximately eight hours on-line at one of the student stations in the 
1500 System. 

The abbreviated CARE 4 content outline which follows in Appen- 
dix C-3 shows only the main topics covered in the course. 



APPENDIX C-2 



STATEMENT OF PURPOSE AND OBJECTIVES 
FOR CARE 4 



The purpose of CARE 4, "Education of Visually Handicapped Chil- 
dren," is to equip regular classroom teachers, particularly those in 
rural areas, with the knowledge and skills they need to manage the 
instruction of visually handicapped children, partially seeing or blind, 
in their classes. The course material has been selected with rural 
classroom teachers in mind since educational services for visually 
handicapped children in rural areas at present are minimal or nonexist- 
ent in Pennsylvania and in most other states. 

At the completion of CARE 4, students should be able to demon- 
strate their abilities to: 



1. 


IDENTIFY 


educationally relevant characteristics of visually 
handicapped children. 


2. 


CONSTRUCT 


instructional objectives for these children. 


3. 


SELECT 


suitable media and materials for instruction. 


4. 


ARRANGE 


proper classroom environmental conditions. 


5. 


DESIGN 


instructional procedures to facilitate learning. 


6„ 


UTILIZE 


appropriate techniques for evaluating the perform- 
ance of visually handicapped children. 
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APPENDIX C-3 



ABBREVIATED CARE 4 CONTENT OUTLINE 



INTRODUCTION 

A. Welcome 

Bo Purpose of CARE 4 

C. Sensory Input Channels 

D. Organizational Patterns for Instruction 

1. Kinds 

2. History of educational programs 

3. Trends in placement 

4. Pennsylvania programs 

E. CARE 4 Objectives 

1. Identify educationally relevant characteristics of visually 
handicapped children. 

2. Construct instructional objectives for these children. 

3. Select suitable media end materials for instruction. 

4. Arrange proper classroom environmental conditions. 

5. Design instructional procedures to facilitate learning. 

6. Utilize appropriate techniques for evaluating performance of 
visually handicapped children, 

IDENTIFICATION OF EDUCATIONALLY RELEVANT CHARACTERISTICS OF VISUALLY 
HANDICAPPED CHILDREN 

A. Identifying Children 

1. Visually handicapped children 

2. Descriptions 

B. Kinds of Information about Children 

1 . Family information 

2. Diagnostic information 

3. Visual acuity 

4. Visual functioning 

C. Collecting Educationally Relevant Information about Children 

1 . Refer to CARE 1 

2. Emphasis on direct observation - purpose, conditions* focus 

D. Characteristics of Typically Visually Handicapped Children 

E. The Process of Seeing 

F. The Human Eye 

Go Common Causes of Limited Vision in School Children 

1 . Types by etiology 

2. Results of visual impairment in terms of visual functioning 
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III. CONSTRUCTION OF INSTRUCTIONAL OBJECTIVES 
A. Definition of Instructional Objectives 
Importance 

C. Kinds - cognitive, affective, psychomotor 

D. Sources of Instructional Objectives 

1. Observation of student performance 

2. Curriculum 

3. Special needs of visually handicapped students 

IV. SELECTION OF INSTRUCTIONAL MATERIALS 

A. Factors to Consider 

B. Materials - books and equipment 

C. Criteria for Selection of Equipment 

D. Criteria for Selection of Printed Materials 

E. Attitudes Toward Special Materials 

V. ARRANGEMENT OF CLASSROOM ENVIRONMENTAL CONDITIONS 
Ac Factors 

1. Light 

2. Noise level 

3. Temperature 

4. Furniture 

5. Space - storage, work, moving about 

6. Equipment 

7. Learning task for particular student 
B. Arrangements for Amy, age 10, grade 5 

1. Information available 

a. Instructional task 

b. Materials to be used 

c. Visual acuity 

d. Family infomation 

e. Visual Functioning 

f. Other sensory deficits 

g. Achievement 

h. Intellectual ability 

2. Decisions to be made 

a. Seating 

b. Furniture 

c. Supplementary lighting 

d. Location of work area 

VI. DESIGN OF INSTRUCTIONAL PROCEDURES 

Application segment for practice using rules and principles 
learned so far and for analyzing key elements of situations 
given and evaluating decisions made in particular given 
situations. 

VII. UTILIZATION OF APPROPRIATE TECHNIQUES FOR EVALUATING PERFORMANCE 
A. Testing 

1. Kinds 

2. Value 

3. Considerations 
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B. Test Materials 

1. Alternatives to regular print tests 

2. Alternatives to writing test answers 

C. Test Administration Procedures 

1. Time limits 

2. Limitations 

3. Alternatives 

4. Teacher attitudes toward evaluation 

D. Use of Test Results 

1. Guide in making instructional decisions 

2. Source of information about present performance 

VIII. SUMMARY AND FINAL EXAMINATION 

A. Summary 

1 . Review of purpose 

2. Review of several main points 

a. Learning behavior 

b. Student characteristics 

c. Attitudes and their potential influence 

B. Final Examination 
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RATIONALE FOR CARE 4 

Recently it was reported that approximately 3.75 million of the 
estimated six million handicapped children in the United States are not 
receiving the special education services they need. Many of these 
handicapped children live in sparsely populated areas where school dis- 
tricts are too small to provide programs for even those handicapped 
children already identified. In addition, trained personnel are fre- 
quently not available for existing programs in rural areas, and inserv- 
ice regular classroom teachers may not possess adequate knowledge to 
make appropriate instructional decisions for handicapped children. 

One group among these handicapped children are those with 
limited vision, both partially seeing and blind. Many of these chil- 
dren can function well in regular classes with a minimal amount of 
direct service from a specialist, if the regular classroom teacher and 
school district personnel are adequately informed. Adequate and acces- 
sible inservice teacher training which allows for individual differ- 
ences in background and teaching experience, in teacher performance, 
and in teacher availability for time for inservice coursework presents 
a challenge to those wishing to improve the skills of inservice regular 
classroom teachers. 

A feasible solution to the problem of providing inservice 
teacher education that is accessible to teachers in rural areas and that 
is responsive to individual differences among teachers would seem to be 
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computer-assisted instruction (CAI). In a tutorial CAI system the com- 
puter selects sequences of instruction which are appropriate to an 
individual's background knowledge of the course content, his rate of 
progress through the materials, and the types of errors the student 
make J as he interacts with the system. 

The goal of the Computer Assisted Remedial Education 4 (CARE 4) 
project is to design a course ent.tled "Education of Visually Handi- 
capped Children" which will contribute to the inservice education of 
regular classroom teachers, particularly those in rural areas, who can- 
not return to a college campus for on-going training. Course content 
is to be selected for its appropriateness for regular classroom teachers 
who have or will have visually handicapped children enrolled in their 
classes. 

CARE 4 course materials are to be prepared by course authors at 
the University of Pittsburgh and programmed for use in the IBM 1500 
Instructional System by the staff of the Computer Assisted Instruction 
Laboratory at The Pennsylvania State University. Once developed, the 
course is to be made available to teachers in a custom-built expandable 
van operated by The Pennsylvania State University CAI Laboratory. The 
van contains 560 square feet when expanded and houses fifteen student 
stations or consoles at which inservice teachers and other education 
personnel take courses. The van with the CAI system can be hauled by 
diesel tractor to remote pans of a state and set up for instruction in 
a short period of time. The mobile CAI system is located for approxi- 
mately six- to eight-week periods on a location adjacent to centrally 
located school buildings. Teachers and other interested persons drive 
to the central location in the afternoon or evening at their convenience 



to take the course on an individual basis. The Mobile CAI Laboratory 
Can serve more than 200 persons during each six- to eight-week stop. 

It is the hope of the project staff that CARE 4 will benefit 
regular classroom teachers and result in more visually handicapped chil- 
dren being able to remain in their local communities for their 
education. 



APPENDIX F-1 



ERIC 



INTERVIEW QUESTIONS 

1. Was your student partially seeing, totally blind, or functionally 
blind? 

2. For what subjects and for what grade did you have the student? 

3. How did he function in reading, travel, and identification of 
objects and people? 

4. How frequently did your student and/or you receive services from the 
itinerant teacher? 

5. What functions of the itinerant teacher did you find most helpful? 
least helpful? 

6. Do you remember any particular worries or concerns you had prior to 
your first day in school with the student? 

7. How would you evaluate your experience with the student? 

8. If you had the opportunity to offer advice or suggestions to another 
classroom teacher, what would you consider important to say? 

9. Were special materials provided for your student? What kinds? Were 
they adequate? What else would have been useful? 
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APPENDIX F-2 



INTERVIEW REPORT 

#1 and #2 

1. Was your student partially seeing, totally blind, or functionally 
blind? 

answer: could see in one eye, but only peripheral vision 

2. Grade: 

answer: grades 1 and 2 

3. Function: 

answer: travel - took taxi to school; little participation in play- 
ground activities 
reading - low level if any at all 
identifying objects and people - no answer 

4. Frequency of itinerant service 

answer: grade 1 - each day 

grade 2 - three times a week 

5. Functions of itinerant teacher most helpful and least helpful: 
no answer 

6. Concerns prior to first day with student? { 
no answer 1 

7. Evaluation of experience with visually handicapped child? | 

i 

answer: frustrating | 

8. Advice for other regular classroom teachers: | 

answer: patience I 

concern for attitudes of children toward visually handi- | 

capped child I 

have materials ready on time I 

expect frustration I 
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#1 and #2 - continued 

9. Special materials provided: 

answer: books and special teaching "for what it was worth" 



COMMENTS: 

Both teachers felt frustration in having student in with the 
regular children when she could and would do little for herself, such as 
pull her seat up to the board. She demanded too much individual atten- 
tion when considering the demands of the other children. Both agreed 
another child who did not have the additional problems might work out 
more satisfactorily. They suggested that a full-time aide would have 
helped. 



117 



INTERVIEW REPORT 

#3 

1. Was your student partially seeing, totally blind, or functionally 
blind? 

answer: had light perception 

2. Grade: 

answer: special class for physically handicapped children 

3. Function: 

answer: travel - in wheel chair 
reading - low 

identification of people and objects - no 

4. Frequency of itinerant service 
answer: daily 

5. Function of itinerant teacher most helpful and least helpful: 

answer: most helpful - obtained Talking Book, flash cards, print 

and braille books, got cassettes which all 
used 

least helpful - no answer 

6. Concerns prior to first day with student? 

answer: no particular concerns; helped to know that itinerant 
service would be available 

7. Evaluation of experience with visually handicapped child? 
no answer 

8. Advice for other regular classroom teachers: 

answer: It helps to know that help is available. Blindness is or 
can be only one problem among many. Watch for verbalism; 
child may not have the real experiences to give meaning to 
words. 

9. Special materials provided: 

answer: Talking Book, cassettes, corduroy blocks, clock 



#3 - continued 

Were they adequate? 

answer: especially the clock 

What others would have been useful? 

answer: textured flag, money glued down, canvas animals 

As for borrowing from the IMC, two weeks isn't long enough 
to use the items. 



119 



INTERVIEW REPORT 

#4 

1. Was your student partially seeing, totally blind, or functionally 
blind? ^ 

answer: partially seeing 

2. Grade: 
answer; grade 1 

3. Function: 

answer: travel - no answer 
reading - no answer 

identification of people and objects - no answer 

4. Frequency of itinerant service 
answer: only for tutoring 

5. Function of itinerant teacher most helpful and least helpful: 

answer: most helpful - visual perception work 

least helpful - tutoring for classroom work 

6. Concerns prior to first day with student? 
answer: mostly about availability of materials 

7. Evaluation of experience with visually handicapped child? 

answer: Frustrating if materials were not available; otherwise a 
great experience for the teacher and the children. 

8. Advice for other regular classroom teachers: 

answer: I would have to know about the child. Providing materials 
were available, it would be a great experience. 

9. Special materials provided: 
no answer 

a. Were they adequate? 
no answer 

b. What others would have been useful? 
telephone number of teacher 



#4 - continued 



COMMENTS: 



Both the teacher and the principal made little distinction 
between children with visual impairment and those with visual perception 
problems. 



APPENDIX G-1 



NOTE TO EVALUATORS 



TO: 
FROM: 



You have been asked to participate in the formative evaluation 
of CARE IV, a CAI course entitled "Education of Children with Limited 
Vision." The course is being designed for presentation to regular 
room teachers in rural areas who may have partially seeing or blind stu- 
dents in their classes. 

As an evaluator, your task is to 'take' the course and to com- 
ment on the content , s equence , quality , and any other aspect of the 
course you think appropriate. Remember, the course is the object of the 
evaluation, not you. Your responses to questions in the course, which 
will be recorded by the computer for us, along with your Evaluation 
Cards and those of the other evaluators will assist us in revising 
CARE IV during the early stages of its development. 

Use one of the Evaluation Cards you receive for each CRT, audio 
message, or image on which you wish to comment. YOU ARE NOT EXPECTED TO 
COMMENT ON EVERY CRT, AUDIO MESSAGE, OR IMAGE - only those which you 
would like to call to our attention either because they are particularly 
effective or because they need revision. 

On each card you use, beside FRAME LABEL fill in the alpha- 
numeric code you see displayed in the lower right hand corner of the 
CRT. If you are referring to an image or audio message, use whatever 
frame label shows while the image is visible or while the audio message 
pi ays . 

The items listed on the Evaluation Cards are suggestions upon 
which you can base your appraisal. Feel free to make additional remarks 
in the space provided. 

When ycu have finished, please return your cards to the proctor. 
Thank you very much for your help. Take CARE! 



MEW/sp 
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APPENDIX H-1 



QUESTIONS FOR EVALUATION OF OFF-LINE MATERIALS 



1. Does the content of the Handbook relate to the purpose of CARE 4 as 
stated on page 2 of the Introduction? 

2. Does the content of the Handbook provide what is necessary for a 
student to meet the course objectives as stated on page 2 of the 
Handbook ? 

3. Does the content of each chapter actually deal with the topic as 
expressed in the chapter title? 

4. Are any Handbook sections confusing or unnecessarily complicated? 
If so, where? 

5. What would you like to see included or omitted in a revision of the 
Handbook ? 

6. What attitudes, if any, do you find reflected in the Handbook con- 
tent regarding: 

a. persons with visual handicaps 

b. regular classroom teachers 

c. school placement of children with visual handicaps? 

7. Refer to page iii for the coordination of CARE 4 Handbook chapters 
with Teaching About Vision and the pamphlet entitled "Helping the 
Partially Seeing Child in the Regular Classroom." 

a. Do the Handbook and Teaching About Vision chapters actually 
fit together or complement each other? 

b. Are there any contradictory statements/ 

8. Please make any additional comments you feel would be helpful. 
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APPENDIX H-2 



EVALUATION OF CARE 4 HANDBOOK 



1. Does the content of the Handbook relate to the purpose of CARE 4 as 
stated on page 2 of the Introduction? 

The purpose of the Handbook, is to provide knowledge and skills 
necessary to deal with visually impaired children in the classrooms. 
The Handbook readily meets this challenge in terms of providing the 
theoretical knowledge. I am however, not quite sure that adequate 
guidelines in terms of providing necessary skills receive sufficient 
.attention. 

The types of skills necessary for teachers of the visually 
impaired children as applied to the purpose of the Handbook should 
receive further consideration. The necessary skills should be spelled 
out to enhance easier reading and practical application in a classroom 
situation. 

2. Does the content of the Handbook provide what is necessary for a 
student to meet the course objectives as stated on page 2 of the 
Handbook? 

If knowledge and skills are synonymous, I would contend that 
the content of the Handbook provides what is necessary for a student to 
meet the course objectives. It seems to me, however, that knowledge as 
an entity cannot be described as skills. It is the application of 
knowledge that I consider as skills. To really meet the objectives as 
stated in the Handbook, some kind of real life situation should be 
included in the course activities, 

3. Does the content of each chapter actually deal with the topic as 
expressed in the chapter title? 

The content of each chapter clearly and relevantly deals with 
the topic as expressed. The content of each chapter gives direct, pre- 
cise accounts that I believe are well organized and should be helpful to 
students who take the course. 

4. Are any Handbook sections confusing or unnecessarily complicated? 
If so, where? 

As I indicated in question three above, the Handbook is orderly, 
well organized, and I do not see any confusing or complicated sections 
of the book. 
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5. What would you like to see included or omitted in a revision of the 
Handbook ? 

The Handbook proposes that teachers who are responsible for 
making decisions regarding the selection of instructional materials con- 
sider three factors (p. 32) I would like the fourth factor to be 
included in making a revision of the Handbook 

The fourth factor that 1 would like you to consider is: The 
particular INTEREST of the group of students should not be neglected 
and should be considered in making decisions regarding selection of 
instructional materials Although, literature reveals that the psycho- 
logical feelings of the visually impaired have no significant deviation 
from the average population, I would, however, like a chapter on the 
psychological effects of blindness, if any, to be included in the Hand- 
book. I would also like to see summaries of the salient points in each 
chapter of the Handbook made in the process of future revision- 

6. What attitudes, if any, do you find reflected in the Handbook con- 
tent regarding: persons with visual handicaps, regular classroom 
teachers, and school placement of children with visual handicaps? 

The Handbook deals more specifically on methods or procedures 
of acquiring knowledge and skills useful to teachers in dealing with 
visually handicapped children rather than on attitudes. The Handbook 
content does not in my view present any specific attitudes outside the 
realm of knowledge and skills development. 

7. Refer to page iii for the coordination of CARE 4 Handbook chapters 
w^tfi Teaching Ab out Vision and the pamphlet entitled "Helping the 
Partial ly Seeing Child in the Regular Classroom." Do the Handbook 
3"° Teaching About Vision chapters actually fit together or comple- 
•nent each other? Are there are contradictory statements? 

A thorough examination of the Handbook, and Teaching About 
Vision and even the pamphlet clearly reveals that the three books fit 
together beautifully. Teaching About Vision , however, contains more 
technical terminologies than either of the other two books. There seems 
to be lack of complement in the sixth chapter of the Handbook and the 
sixth chapter of Teaching About Vision - Chapter six in the Handbook 
deals with the design of instructional procedures while chapter six in 
Teaching Ab out Vision seems to dwell on the problems encountered in 
reading as a t^esult of poor vision 

8. Please make any additional comments you feel would be helpful. 

As I stated in my view to question two, the knowledge and skills 
that the Handbook intend to impart to students will be more effective 
and meaningful if some kind of practical application goes with the use 
of the book. This may be done by having the students do some demonstra- 
tions or dramatizations of the contents of each chapter or actually work 
with the visually impaired children for a short period of time to sup- 
plement the factual contents of the Handbook- 
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APPENDIX I 

AUTHOR FORMAT SHEETS 
IMAGE PLANNING FORM Imacie No. 

Date: Author: Date Required: Course: 

SPECIAL INSTRUCTIONS: 



If lettered : 

[] Black letters on white ground 
ij White letters on black ground 

I] Colored letters 

Colored qround 



If. drawn : If Photographed : 

[] Line drawing Descriotion: 

[] Area drawing 

[1 Other 



SKETCH 



PfeoAe oAc the 4pacc enclosed by doUai lint6 ioK ttjpamAJXzn copy. 

^i4e tha toAgz rectangle io^ dyxiainQ {MbnvnbuA to use teXteA^ no hmaJUoA than the 

rru.iujnum Uzz* In the tnhqe KtcXanqlv,, and typejAVUtten Uze In the J^maJUi oAca). 

THIS SIDE up: Horizontal Format Must Be Used for Images. THIS SIDE UP: 




*Minimum letter sizg^for use in 
large rectangle = 
2/72 
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APPENDIX J-1 



COMMENT SHEET 



Evaluator 01 

1. Rationale .— OK - although can bs polished. Your first paragraph, 
which is prominent because it is first, ought to be improved - espe- 
cially the last sentence. I'd like to see you convey more "com- 
passion" for the real issues facing rural areas in getting and 
keeping qualified teachers, since your statement now sounds more 
like an indictment! Also, I think the last half of the sentence 
refers to regular teachers who have been assigned some handicapped 
children along with their "normal " children, but the present state- 
ment is a bit muddy. 

2. Purpose and Objectives .— general Iv OK. Perhaps one minor point 
should be mentioned for your consideration. You've approached the 
problem from the standpoint that the regular classroom teacher can 
be helped through CAI to be much more effective in her day-to-day 
dealings with the VH child. Tm wondering though if somewhere in 
the presentation we should not also convey the idea that there are 
resources available to help her, that she isn't going to be com- 
pletely out on a limb if she makes a wrong decision, Do we need to 
give status to this aspect by adding another objective "Seek help 
from appropriate resources?" The course presents material and 
equipment resources What about other kinds of resources, particu- 
larly for instruction? Or do you feel this would be covered suffi- 
ciently elsewhere? 

3. Content .— OK. unless you want to pursue above suggestions. I do 
have a few suggestions or comments for your consideration: 

A. Vision Screening .— Do you differentiate between vision 

screening findings and an eye report? Today a school nurse, 
or a lay volunteer from PAB or Junior League or a remedial 
reading teacher armed with a telebinocular can give a class- 
room teacher a visual acuity and "diagnosis." You assume the 
teacher has available to her an eye report. Despite your 
cautioning on educational decision making based exclusively 
on diagnosis, perhaps the teacher will be further misled by 
information from screening ! (And especially in rural areas 
with limited medical staff.) 

3- Teacher AUitudes and Parental Attitudes .--and child's own 
attitudes seem properly stressed. I'd like to see reference 
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to peer group attitudes and how some children can become 
favorably disposed toward themselves or to the use of appro- 
priate materials or equipment because of the interest and 
enthusiasm of their fellow students. 

C. Sources of Materials , --Should you not also include in your 
listing "state IMC's or repositories?" Many states are 
developing these now. 

D. Criteria for Selection of Materials —Does size include bulk 
and weight? What about things that are too small to see - 
would a model help or confuse? Or did you deliberately avoid 
this general issue? 

E. Criteria - Print .— In type size and style do you speak to 
type spacing? What about binding, durability? What about 
dirty pages that cut down on contrast? What about books 
delivered unbound ? 

F. Environmental Factors — Don't forget the dirty chalkboards 
which also reduce contrast. 

6. Do you want to bring out somewhere the legal responsibility 
of school districts to provide for their children? I don't 
think we should convey the idea only that VH children can be 
successful and that the teacher or district is being kind or 
tolerant or altruistic or thrifty by taking these youngsters 
into their system! Perhaps a suggestion of the VH child's 
legal right and the district's legal responsibility should be 
included in the presentation. This is not in the outline - 
although you have brought these points out in the chapters I 
didn't review. 



Ho Dr you need any section devoted to how regular class teachers 
work with the itinerant teacher, with 0 & M personnel? 

L Did you speak to PROGNOSIS as wsU as DIAGNOSIS? Many people 
are thrown by the "PROGNOSIS - POOR" statement in eye reports 
as they don't know what this means 

4. Test .— Quite good- I especially like those questions which present 
a hypothetical child and demand a solution. Only one suggestion 
for 2-c, Since w^ no longer have Supervisor of Special Ed. and this 
v«s written fo" ennsylvania especially, suggest you change to 
"Special Educa^.jn Director." 

I think the content is covered well. You wouldn't want it 
any longer, and I think you hit the high spots, 

5. To me the only test that would .ell if the teacher had really 
mastered the contents would be how well he would respond to a real 
live VH child in his classroom! It's like writing objectives for a 
federal project; one finally learns how to write them acceptably. 



but actually translating into a dynamic educational program is some- 
thing else. I think the test at least THEORETICALLY tests the 
teacher's mastery of objectives. Why don't you do a POST DOCTORAL pro- 
ject on those who take the course and then try to apply their skills? 



APPENDIX J-2 



COMMENT SHEET 



Evaluator 02 

1. Rationale .— Excellently presented. The rationale is excellently pre- 
sented; ^However, I would wish that it could be expanded to include 
city teachers whose service from trained personnel may be limited 
because of time factor and also all teachers who may be the first 

to identify a visually handicapped child or to prevent some child 
from being labeled as visually handicapped who is not. 

Of particular concern to me are the children whose doctors or par- 
ents do not wish them "labeled" or given large print with the 
result the child suffers educationally or at the other extreme, the 
child who corrects to almost normal vision but whose parents can 
only accept the vision uncorrected with the result that the teacher 
is persuaded to make special adjustments. 

In addition, should the rationale be expanded to teach the differ- 
ence between vision and perception? 

2. Purpose and Objectives .— Certainly met. Are there appropriate 
school personnel though to whom a teacher may make referrals to 
state or private agencies for support services other than educa- 
tional materials, who can advise on long term career planning? For 
many visually handicapped students, an educational plan is inade- 
quate unless there is support from outside agencies. 

3. Course Outline . --Content more than covers purpose 

a. Visual Acuity Measurement - Do you find Allen Care measure- 
ment or optometric, both of which must be adjusted. 

b. Definition - Partial sight acuity definition of 20/70 is 
eliminating many whose visual functioning is lower but acuity 
higher. This definition needs to be broadened. How about 
20/50 high, high myopes, cataracts, or beginning of degenera- 
tive diseases? The functional definition is excellent and 
the summary is well taken. 

c. Injuries - Scratched by branch is unbelievably high in 
reporting. 
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d. Low Vision Aids - 

1 . Factors limiting use - 

a. Movement distortion with distance telescopic aids 
b Motivation of chi id 
c. Acceptance by peers 
An excellent course outline. 

4. Final Exam .— Well prepared. It does test on course material and 
requires thinking as well as memorization. 

I note that the course outline does not include some categories of 
children who might be considered visually handicapped. Perhaps 
they are mentioned in the material. 

Temporary consideration might be given to the following: Child 
with patched eye, amblyopic eye, may be 20/70 or less The smart 
patched eye child solves his problem by removing patch or changing 
It to the other eye, both of which defeat the purpose but keep him 
seeing. 

The recently enucleated or severely injured eye may constitute an 
educational problem temporarily until adjustment is accomplished. 
It might be that the teacher trained in visually handicapped prob- 
lems would be able through understanding to prevent future emotional 
problems. 

I hope that the above comments are adequate. It was with consider- 
able Interest that I perused the prepared material and I was 
Impressed with the amount of valuable information that has been 
included in the course. It is my hope that this material can be 
shared with other areas of the country. 

Thank you for allowing me to see the material. I wish I could take 
the course. 



APPENDIX J-3 



COMMENT SHEET 



Evaluator 03 

1. Rationale —Does indeed seem logical. Should the following be men- 
tioned, in this section or elsewhere? 
length of course 
hours of credit 

who eligible (wish there could be a modified version of such a 
course, especially for parents!) 

2- Purpose and Objectives , -Is "manage the instruction" the best term 
1" this section - is "effectively teach" more positive (or some- 
thing similar)? This terminology recurs in Handbook also (e.g., 
p. 11). 

Could "positive communication with parents" be mentioned, somehow, 
within the six good points which serve as central theme throughout 
all aspects of this program? 

Could "relating to available appropriate resource people" be added, 
suitably, to this section? (See also Handbook , p. 2.) 

Generally, purpose and objectives do meet the need presented in 
the rationale. 

3. Course Out1 ine . --Concerning course outline, in particulate, may I 
suggest several specifics or raise several questions: 

Chap. I^re: 

A.-l. What does this mean? Does it indicate gearing the 

course to students' needs? 

"deal effectively" or "effectively teach"? 
D.-5.a Add "... and relevant local resource personnel"? 
E= Could involvement of appropriate resource personnel be 

meaningfully added to one of the existing categories? 



The items identified in the course outline by this evaluator 
cannot all be located in the abbreviated course outline reported in 
Appendix C-3 because this evaluator examined a more detailed course out- 
line to make her comments. 
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Chap. II re: 

A. -2.b. Would improvement of functional vision thru develop- 

mental use be well added here? 

B. -3. Would 'E' added here reiterate the above? 
-4. Should orthoptist be added here as 'd'? 

Chap. Ill re: 

C- Could "social" be fittingly added here as #4? 

Chap. IV re: 

D.-6. above weighed in terms of readability for individual 
youngster? 

Chap. V re: 

Before A- Testing should there be a brief presenta- 
tion of other evaluation procedures of weight - home- 
work, oral participation? 

Should there be mention of involvement with parents of 
visually impaired children? 

Inasmuch as mobility is an area calling often for 
special helps should more be mentioned in this regard - 
and does Pennsylvania have relevant resource people 
" available? 

In answerto the specific question re: #3, the course content does 
relate to the expressed purpose and objectives. 

4- and 5. Final Exam — It does seem to cover the course content. May 

I suggest the following: 

Chap. 4, question No. 3 - Could this be broadened to reflect the 
Printing House's many other functions and available materials? 
(This would call for more such information in the course itself.) 

Chap. 5, questions no. 5 and 6 - More of this kind of question 
would seem to me to be especially valuable. 

Other Comments ,— 

Is it possible to request feedback re: course's practicality 
after students have put their knowledge to work for a 1/2 to a year 
period? 

Are there sufficient teachers already involved with visually 
impaired youngsters to allow a "big brother" sort of arrangement for a 
year's time, once a student completes the course? 



It seems to me its content quite well relates to the purposes 
and objectives of CARE 4. In particular regard to question 5 (of 
evaluation form) concerning attitude^, it might be beneficial to peri- 
odically mention where appropriate the words "as with children, gener- 
ally." (Much of what is touched upon in the Handbook is true, cer- 
tainly, not just re: visually impaired children, but perhaps it bears 
saying more frequently.) 

Should there be specific suggestions herein re: how the 
teacher can manage the extra time involved when a visually impaired 
student is in the class (e.g., volunteer help, student 'buddy' system)? 

Should more space be devoted to: 

references (e.g., new book edited by B. Lowenfild?) 
visual functioning (e.g., assessment of APH materials) 
mannerisms 

explanation of terms re: vision (p, 22) 
visual aids 

interpretation of one eye record in further detail (pp. 41, 42) 

mobility 

social needs 

How is it determined into whose class a visually impaired child 

will go? 
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STUDENT-CONTROLLED OPTIONS TO BRANCH 



Chapter Location 



Number of Students 
Choosing to Branch 



CARE 4-1 
ba25a 

CARE 4-2 
cal0a 

cd03a 
ce38a 

ce49a 
ce64b 
ce70b 
ce97b 

cf05a 



Would you like additional informa- 
tion about the history of educa- 
tional programs for blind children? 



Would you like to read a technical 
explanation of the construction of 
the Snellen Chart? 

Would you like to review CARE 1, 
Chapter 14? 

Would you like the definitions of 
any of these words: direction, 
diffusion, radiant energy? 

Do you know the definition of 
refractive medium? 

Do you want to check the definition 
of 'diverge'? 

Do you want to check the definition 
of 'converge'? 

Would you like some more information 
about cataract, glaucoma, and 
nystagmus? 

Would you like to review any causes 
of loss of vision? 



6 
1 

2 
6 
1 
1 

2 
2 
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Chapter Location 
CARE 4-4 
ec02b 

ec04a 

ec26a 

ec33b 



Would you like to hear more about 
larg type and type size? 

Would you like to hear more about 
the work done at APH? 

Do you want to review any of the 
pieces of equipment? 

Do you want the address of any of 
the three regional instructional 
materials centers in Pennsylvania? 



Number of Students 
Choosing to Branch 



4 
7 
0 
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APPENDIX L 



ON-LINE COMMENTS MADE BY 
FIELD-TEST STUDENTS 



VZWC CARE4 

If an attempt is being made to demonstrate the lack of an auditory 
input, you arc succeeding ... The sound portion is almost completely 
inaudible. 

VZWC CC40M 
Prior frame was very confusing 

VWAT LINKER 

The audio portion for Chapter I was inconsistant in quality - sometimes 
it seemed as if the speaker was too far away from the mike and some- 
times he seemed to be too close to It. 

VZWC CARE4 

Re audio portions of this CARE4 lesson, the messages are garbled, low 
volume, and also seem inappropriate to the subject being shown on the 
CRT and viewer. I suggest that they be debugged real fast since this 
course, while short, is more confusing than CAREl. 



DA03A 

RE DA 03. . . . Solve is ambiguous. . , I may solve mentally, 

or I may solve physically- It's a matter of semantics. . Solve vanes 
in whatever context it is put into. 

VZWC EC41A 

Why ask me to answer if you are going to give the answer anyway. I 
assumed all of the criteria were important, I chose the two I (you did 
ask me) felt was most important for the girl. 

VZWC FC08C 

After having gone through Amy's problems once, I figured I could pro- 
gress without all the steps. I am caught in a loop. Help Help Help 

VZWC HA09A 

RE HA09 . . . Computer does not accept some of the answers which were 
given. . i.e. . felt marker. Floyd mentioned this . . , Special con- 
sideration . . . map . . . Please check your program to see if the 
other answers might no be put into it. 
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VZWC CI 2 
Sorry about that. 

C13 

Re last question . . . Average achievement as to ability. 
C4A 

Please note that the subject's vocabulary is limited. Congenital = 
prenatal . . . Accident = injury . . . Tumor = neoplasm . . . Disease 
was used in the general term i.e., collectively = diseases. 

C5 

Your question referred to refractive parts of the eye. Both the 
cornea and the lens are refractive agents in the eye. There is no, 
repeat no, reference to the relative refractive powers of these units 
in either text or handbook. My answer was predicated on the optical 
values of both units. As a professional photographer I made an educated 
guess as to the power of both and assumed the power of the thicker was 
more powerful. A possibility also was that it was in the program, but 
the tapes were fouled up and I may have missed it. This foul-up was 
reported to computer during the lessons. 

E2B 

Re previous question. . . . This happens to be a sore point with me. I 
can get a hell of al«t more by going thru my principal than some of our 
junior directors. I question the validity of your querry. ... It 
depends on circumstances 

VZSS E5 

My answer automatically accepted as the cursor moved off the screen. 
I wasn't finished and would have finished it by saying varying light 
conditions. 

VZWC E5 

Re previous question . , . Your use of an exclusive vocabulary has 
fouled me up. Only one line was permitted in the answer, . . . Not 
stated 1n original question. ... I assumed I'd be permitted to con- 
tinue on the next line. I began by referring to the continual adjust- 
ment due to the focusing of the device. I began to try to point out 
the narrow range with the device but got cut off. 

VZWC E6 .) 

On the last question regarding the limits of the magnifying glass, I i 

proceeded to give the answers gives a limited field and it has a fixed ? 

focus. When I went to push the return button, the answers appeared i 

Will I get credit for my answers. | 

5 

VZSS E7A I 

Slate and stylus (your answer) are the same as stylus and slate (my i 
answer) . 



VZWC E7A 

Third choice was difficult to see. Slide is of poor quality. Also, 
fourth, was correct term a phonograph - talking book? 

VZSS F6 

Why is B the best answer? It seems to me that using the tape recorder 
might be a quieter and more convenient method than the typewriter. 

VZWC G2 
Re G2 . . . Same difference. 

JA 

Please review your programming and the audio portions of your course. 
The audio was extremely difficult to comprehend at times. Tape hard- 
ware program. Also please consider the semantic variations that all of 
us have when taking these tests. Realistically, you cannot include all 
possible answers. But try to have a broader acceptance pattern. The 
workbook was rough ... in all meanings of the word. It needs work, 
both on content and presentation. You really should review it. I 
would be willing to help you as a lay-teacher. I have experience in 
writing, editing, and communications. 
VZWC 

*END 
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FLOWCHART OF EVENTS AT CAI LABORATORY 




Event Identification 



lo Authored material submitted to programer 

2o Authored material prepared for keypunching 

3. Authored material keypunched 

4« Card deck assembled and listed 

5. Initial debugging completed 

6, Image planning forms sent to graphic artist 
7o Slide processing completed 

8o Image preparation completed 

9o Audio messages edited 

lOo Audio messages typed 

IK Author review completed 
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[13) > C14) > 05 
4 ^ 1^ 

hrs. mo, wks. 



15 hrs. 



hrs. hrs. hrs« hrs. 



E vent Identlfica'M (Continued) 



llo Author review completed 

12„ First revision of images completed 

13» Images photographed 

14. Answer print received 

15c Image copies received 

16. First revision audio messages typed 

17. Audio messages recorded 

18o Student audio tape completed 

19o Audio recordings debugged 

20. Audio tape copies made 

2K Program revisions and debugging completed 

22. Students scheduled - 1st group 

23. Students complete trial run 

24. Author review of student records and comments completed 



I 



I 
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1 

mo 



30 hrs, 



2 

wks. 



hrs« hrs„ hrs„ hrs. 



Event Identification (Continued) 



24 „ Author review of student records and comments completed 

25« Second revision of images completed 

26o Images photographed (or errata noted) 

27 o Answer print completed 

28o Copies of image reel received 

29» Second revision of audio prepared 

30o Audio messages recorded 

31. Student audio tapes completed 

32. Audio messages debugged 

33. Audio tape copies made 

34« Program revisions and debugging completed 

35o Students scheduled - 2nd group 

36. Students complete trial run 

37. Author review of student records and 
comments completed 
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hrs. hrSo hrSo hrs« 



Event Identification (Continued) 



37« Author review of student records and comments completed 

38. Final revisions of images completed 

39. Final revisions of images photographed 
40o Final answer print approved 

4K Image copies received 

42. Final version of audio text prepared 

43. Final version of audio messages recorded 

44. Student audio tapes completed 

45. Audio debugging completed 

46. Audio tape copies made 

47« Program revisions and debugging completed 

48o Preparation of course completed 



